Chapter 4: Statistical Predictive Models

Chapter Introduction

Chapter 2 established the data-engineering contract and Chapter 3 examined what the resulting statistics justify. This chapter connects prediction with delivery promises, inventory demand, product margins, customer inactivity, fraud investigations, marketing offers and financial decisions. The business objective comes first: define the observation unit, Y, horizon and outcome-availability time. Then distinguish the training loss, the metric used to select a model and the rule that turns its output into an action. These choices can change the preferred model or decision even when the input rows do not change.

The small business comparisons below are invented teaching illustrations, not evidence of measured deployment gains. A better aligned objective still needs later evaluation against a baseline, with costs, capacity, uncertainty and complete follow-up.

Our public nine-stock handoff is deliberately small and serves as the financial transfer case. The regression target is next-month simple return. The binary event is a closing-to-closing next-month return of −5% or worse. This definition must remain separate from intramonth drawdown, volatility and a worst-day extreme. The same feature table supports both questions, but their losses, probability statements and evaluation metrics differ.

The first fitting clock is 1 January 2021, and model/rule selection ends at 1 January 2022. A label must have arrived strictly before the relevant clock. The split therefore contains 207 training rows, 99 validation rows, 306 later evaluable rows and nine pending outcomes. Eighteen boundary rows are excluded from these partition-specific samples because their labels arrive after the respective freeze. Later annual refitting may legitimately admit them once they become available. Every denominator follows a stated timing rule.

Download the worked Chapter 4 notebook. The slides provide a complete interactive self-study route. This companion supplies additional mechanism, interpretation and failure modes. Run cells in order within a section, wait for each result, modify one input and explain the difference. Classroom discussion selects important mechanisms from the full material; it does not limit the assigned conceptual coverage.

Sections 4.1–4.3 build the statistical toolkit. Sections 4.4 and 4.5 retain the full causal-design and attention-feature extensions. Section 4.6 connects predictions to a fixed portfolio policy and a reconciled net account. A weak test result is a finding to explain, rather than a reason to keep tuning the same holdout.

How to use the numbered learning path

Follow the numbered parts in order within each section. Each part states a question and a result. Read the concept, predict the example, run it, and explain the output before attempting the check.

Classroom discussion selects important examples and experiments. The complete Slides and Book support self-study.

Worked examples run automatically when the page opens or refreshes. Each cell includes required earlier imports, data and definitions under Previously introduced setup. Run Code includes that setup; Copy complete code copies the whole cell. Practice and repair cells run when you choose. Make input changes in the current cell before the operation you are testing.

§4.1 Targets, Regression and OLS
§4.2 Inference and Model Selection
§4.3 Nonlinear Models and Loss Classification
§4.4 Causal Questions and Research Designs
§4.5 Cross-sectional Attention Features
§4.6 Walk-forward Decisions and Portfolio Accounts

Targets, Regression and OLS

Part	Question
4.1.1 Targets, rows and information clocks	What can the model know and what must it predict?
4.1.2 Fitting and interpreting OLS	What coefficient minimises squared residuals?
4.1.3 Assumptions, baselines and forecast errors	Which claim is supported by this regression?

Targets, rows and information clocks

Question: What can the model know and what must it predict?

Result: A sample that retains stock/date columns and separate decision, availability and target times.

Begin with the definitions and assumptions; then trace the worked example.

Why the target definition comes before the model

The conditional mean and the event probability summarise different features of a conditional distribution. A positive mean can coexist with a substantial probability of a severe loss. A classifier reporting a low loss probability says nothing about the magnitude of the remaining upside or of losses beyond the event threshold. Neither output alone specifies a portfolio. The statistical target, the economic decision and the accounting object must remain distinguishable.

The squared-error-optimal prediction is the conditional expectation when it exists. Absolute-error loss targets a conditional median. Bernoulli log loss targets the conditional event probability under a well-specified probabilistic model. Choosing a loss is therefore part of choosing what to estimate, rather than a cosmetic setting in a Python class. The later tree and boosting examples inherit the same distinction.

The exported momentum and volatility columns are already engineered teaching features. Chapter 2 explains identity, windows, sector transformations and availability. Here we inspect those meanings instead of treating every numeric column as a candidate predictor. A label-status flag can be numeric and still be unsuitable for X because it encodes knowledge about a future outcome. A previous model prediction needs its own training provenance before it can be used as a feature.

Define the delivery outcome before choosing a model

For the same four deliveries, does predicting minutes answer the same question as predicting a missed promise?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

Define the delivery outcome before choosing a model · Python

One row is a delivery. Minutes is a continuous Y; late means actual minutes strictly exceed the stated promise. Mean minutes, event probability and a high time quantile answer different questions. Use order-time features; actual delivery time is an outcome, not a predictor.

One table, two statistical questions

Regression: What is the next-month simple return $Y_{i,t+1}$ conditional on today’s features?

Classification: What is the probability of $Y_{i,t+1}\le-0.05$? This event means a next-month closing-to-closing loss of at least 5%. It does not mean an intramonth 5% drawdown.

The public Chapter 2 handoff supplies nine stocks, 71 signal dates and 639 candidates. This small current-survivor sample teaches statistical reasoning, rather than validating a profitable strategy.

The same questions arise in demand forecasting, equipment failures and insurance: predict a continuous outcome or the probability of a defined adverse event.

The modelling path preserves the data contract

%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 20}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
    A["aligned available<br/>features"] --> B["Time-ordered<br/>train / valid / test"]
    B --> C["OLS<br/>next return"]
    B --> D["Logit<br/>P(loss ≥ 5%)"]
    C --> E["Evaluate fit<br/>and uncertainty"]
    D --> E
    E --> F["Freeze<br/>decision rule"]
    F --> G["Walk forward<br/>reconcile accounts"]

Keep stock/date columns, feature availability and label maturity visible through every stage. A missing future label prevents fitting or evaluation, but does not remove the current prediction candidate.

Load the case and freeze the three clocks

The split prints 207 training, 99 validation, 306 test and 9 pending rows. Freeze fitting at 1 January 2021 and rule selection at 1 January 2022. December 2020 and December 2021 each contribute nine boundary rows whose labels arrive after the respective cutoff.

The table’s stored predictions are ignored. We fit new models using only the three named features and two sector dummies. Future returns, label status and prior predictions never enter X.

Read the feature matrix before fitting

Input	Meaning	Unit
`ret_12_1`	Twelve-to-one momentum	Simple cumulative return
`rvol_21d`	Trailing realised volatility	Source volatility units; verify annualisation metadata
`momentum_sector_z`	Momentum relative to same-date sector peers	Standard deviations
Sector dummies	Financial Services and Technology versus Energy	0/1 indicators

An intercept plus every sector dummy is collinear. drop_first=True supplies a reference group. Static sectors and assumed month-end availability remain classroom limitations.

Predict: Why would fitting on every numeric column accidentally admit future information?

Fitting and interpreting OLS

Question: What coefficient minimises squared residuals?

Result: Fit OLS and interpret a conditional association in its units.

Builds on Targets, rows and information clocks.

A different loss can require a different stock level

Demand is 2, 2, 2 or 10 units. Would you order the mean, median or a higher quantile when a shortage costs more?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

A different loss can require a different stock level · Python

Squared loss targets a conditional mean; absolute loss a median; pinball loss at tau a tau-quantile. Here shortage cost 4 and surplus cost 1 imply tau=4/(4+1)=0.8, and order 10 minimises this sample cost. These are illustrative historical outcomes, not a guaranteed future optimum. Sales can be capped by stock: observed sales need not equal unconstrained demand.

Multiple regression separates conditional associations

\[Y_i=\beta_0+\beta_1 x_{i1}+\cdots+\beta_p x_{ip}+\varepsilon_i,\qquad E[\varepsilon_i\mid X_i]=0.\]

$\beta_j$ is the conditional change in expected Y for a one-unit increase in $x_j$, holding the other regressors fixed within the model. Correlated features can have very different marginal and conditional associations.

The residual $e_i=y_i-\widehat y_i$ is observed after fitting. The unobserved error $\varepsilon_i$ belongs to the population model.

A predictive coefficient requires stability to generalise. A causal interpretation requires additional identification assumptions (§4.4).

Population errors, fitted residuals and influence

The population error is the gap between the outcome and the specified conditional mean. A fitted residual uses estimated coefficients and is constrained by the least-squares normal equations. With an intercept, residuals sum to approximately zero in the training sample. That fact is mechanical and supplies no independent evidence for exogeneity, Normality or a correct functional form. Residual structure across fitted values, dates and groups can still reveal failures.

Large residuals and high leverage are different. An unusual outcome can have a large residual without an unusual predictor vector. A row far from the observed feature cloud can have high leverage even with a modest residual. Influence combines both ideas: how strongly would estimates change if the row were removed? Diagnose the row against its original source before calling it an error. Legitimate stress observations often carry precisely the risk information we need.

OLS chooses coefficients by squared residuals

\[\widehat\beta=\arg\min_b\sum_i(y_i-x_i^\top b)^2.\]

With full column rank, $\widehat\beta=(X^\top X)^{-1}X^\top y$. Numerical software uses more stable linear algebra rather than requiring you to form the inverse.

Squared loss heavily penalises large errors. An extreme return can materially influence coefficients. Check leverage, residuals and sensitivity before deleting an observation.

Check: A low training sum of squares establishes fit to observed outcomes; it does not establish future prediction quality.

Fit OLS and compare training with validation

Training R² is 0.1766, adjusted R² 0.1561, while validation RMSE is 0.1014 in return units (10.14 percentage points).

The clustered fit has identical point estimates but different standard errors. It allows within-date correlation across stocks. Only 23 training dates supply clusters, so large-sample inference remains approximate.

Interpret a coefficient in the right units

The momentum coefficient is about −0.0484. A 0.10 increase in momentum corresponds to about −0.00484 predicted return, conditional on the other inputs in this fitted sample.

The Technology coefficient is about +0.0880 relative to Energy, conditional on the numeric features. This is a sample association, rather than a prescription to change a stock’s sector.

Assumptions, baselines and forecast errors

Question: Which claim is supported by this regression?

Result: Separate fit, inference and later-sample prediction against a baseline.

Builds on Fitting and interpreting OLS.

What the fitted regression can and cannot identify

Conditional interpretation means holding the other included regressors fixed within the fitted linear specification. It does not mean that changing momentum physically causes the corresponding return change. Sector dummies change the comparison group. An interaction changes the slope at different values of the interacting feature. Collinear regressors can make coefficient interpretations fragile even when fitted values appear stable. These are reasons to state the model equation beside coefficient output.

The small sample contains contemporaneously correlated stock observations. Treating 207 stock-month rows as 207 independent experimental units would overstate the amount of information. Clustering by date allows arbitrary within-date error covariance in the chosen variance estimator, but depends on having enough suitable clusters. It does not automatically handle every form of serial dependence across dates. Chapter 3’s time-aware resampling provides another diagnostic perspective.

Assumptions have different jobs

Assumption	What it supports
Full rank	Identifiable coefficients
Correct conditional mean / exogeneity	Unbiased or consistent OLS under suitable sampling conditions
Constant error variance and uncorrelated errors	Classical OLS variance formula
Normal errors with fixed-X classical assumptions	Exact finite-sample t/F inference
Suitable temporal stability	Generalisation to future periods

OLS can produce predictions without Normal errors. Robust covariance changes inference, rather than repairing a wrong target, leakage or an unstable conditional mean.

R², adjusted R² and forecast errors

\[R^2=1-\frac{\sum_i e_i^2}{\sum_i(y_i-\bar y)^2},\qquad \bar R^2=1-(1-R^2)\frac{n-1}{n-p-1}.\]

These formulas assume a model with an intercept. Adding regressors cannot reduce training R². Adjusted R² penalises their number, but remains a training criterion.

\[\operatorname{RMSE}=\sqrt{n^{-1}\sum_i(y_i-\widehat y_i)^2},\qquad \operatorname{MAE}=n^{-1}\sum_i|y_i-\widehat y_i|.\]

Evaluate future predictions against a simple training-fitted baseline. Out-of-sample R² can be negative.

A baseline makes model error interpretable

Compare both errors on exactly the same validation rows. A flexible model is useful only if the improvement serves the intended decision and survives future evaluation.

Modify: Restrict the numeric feature list, rerun fitting, and explain which quantities may improve mechanically and which need validation evidence.

Self-check: inference, prediction and timing

Self-check: Which statement is justified by the OLS example?

Training R² proves the signal will make money
A coefficient measures a conditional fitted association, and future forecast quality needs chronological evaluation
Normal-looking features guarantee independent errors
Pending outcomes should be filled with zero before training

Feedback: OLS coefficients describe conditional fitted associations. Prediction, causal interpretation and portfolio profitability require separate evidence.

Repair the regression sample

Select only the declared eligible training rows. The starter currently includes future validation/test dates and pending labels.

OLS does not require Normal predictors

Self-check: Which condition is needed to define an identifiable OLS coefficient vector?

Every predictor is normally distributed
The design matrix has full column rank
The training R² exceeds 0.5
Every outcome is positive

Feedback: Full column rank prevents redundant coefficient combinations. Normal predictors are not an OLS requirement.

Training and validation measure different things

Self-check: Adding an irrelevant regressor to OLS with an intercept can do what?

Reduce training R² mechanically
Leave or increase training R² while worsening validation error
Guarantee a lower validation RMSE
Eliminate residual variance

Feedback: The larger model includes the smaller model as a special case, so its training SSE cannot rise. Future error can worsen.

Inference and Model Selection

Part	Question
4.2.1 Coefficient uncertainty and residual checks	What uncertainty and diagnostics accompany the fit?
4.2.2 Specification, redundancy and regularisation	How do feature choices change bias and variance?
4.2.3 Frozen selection and held-out evidence	When can the final test be opened?

Coefficient uncertainty and residual checks

Question: What uncertainty and diagnostics accompany the fit?

Result: Interpret robust intervals, joint tests and residual patterns.

Begin with the definitions and assumptions; then trace the worked example.

Choose a validation score that reflects the decision

Two forecasts make errors on different products. Which model wins if errors on high-margin items matter more?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

Choose a validation score that reflects the decision · Python

Model B wins unit RMSE; A wins the declared margin-weighted absolute error. Margins must be available when decisions are made. This weighted score is a chosen error penalty, not automatically realised profit. Select the score and model on development data; estimate final performance on a later untouched sample.

Inference after a model search

Classical coefficient inference conditions on a specified model and sampling design. A search that tries transformations, subsets and thresholds before reporting the smallest p-value changes that design. The selected estimate has survived a filter that favours unusually large apparent effects. A conventional confidence interval from that single selected fit does not generally account for the filter. Record the complete search and use a suitable confirmatory sample or selection-aware method when making inferential claims.

Prediction-oriented selection and explanation-oriented inference can choose different models. A correlated proxy may predict well while obscuring an interpretable coefficient. Removing a variable solely because its individual t statistic is small can worsen prediction or change the target association through omitted-variable structure. A joint restriction tests a different null from a collection of separate restrictions. Interpret what each test actually asks before using it to justify a modelling action.

Validation is a finite resource

A single validation score has sampling noise. Selecting the minimum among many noisy scores tends to favour an unusually optimistic result. Even when no future test labels leak into training, repeated human adaptation to the same validation set can consume its independence. Bound the search in advance, retain each attempt and use time-aware cross-validation or nested evaluation where justified. More sophisticated evaluation also needs enough independent periods to be useful.

Our 31-subset demonstration is intentionally small and transparent. It shows how exhaustive search grows as the feature pool expands. Thirty candidate features would create over a billion non-empty subsets, and the statistical selection problem becomes serious long before the computational problem. Screening features with the entire sample and then declaring a later split untouched is still leakage. Screening itself is a fitted operation that belongs inside the training procedure.

Rebuild the case before model selection

This section runs independently. Preserve the same training and validation clocks. Model selection uses validation data; the 2022–2024 test remains unopened until the rule is fixed.

Confidence intervals describe coefficient uncertainty

For a classical homoskedastic linear model,

\[\widehat{\operatorname{Var}}(\widehat\beta)=s^2(X^\top X)^{-1},\qquad s^2=\frac{\sum_i e_i^2}{n-p-1}.\]

A coefficient interval uses estimate ± critical value × standard error. A t statistic tests a specified coefficient value, often $H_0:\beta_j=0$.

A confidence interval for the conditional mean differs from a prediction interval for one future return, which also includes outcome noise. Clustered standard errors address a different error structure and typically use approximate inference.

Read robust coefficient output

State the covariance estimator, clustering unit and number of dates. Robust standard errors do not make the coefficient causal.

Repeatedly testing coefficients after searching many subsets changes the inference problem. A nominal p-value from the selected model ignores selection unless the procedure accounts for it.

An F test asks about several coefficients together

For q restrictions in nested classical OLS models,

\[F=\frac{(SSE_R-SSE_U)/q}{SSE_U/(n-p_U-1)}.\]

The null says all specified restrictions hold. A significant joint test need not imply every individual coefficient is significant, particularly with correlated regressors.

The code uses a cluster-robust Wald test with asymptotic calibration, rather than claiming the exact classical F distribution.

Residual plots test the story behind the fit

A curve suggests a missing nonlinear mean relationship. A widening fan suggests changing error variance. Clusters by date may reflect common shocks. None can be ruled out by a single global residual mean near zero, which OLS with an intercept enforces mechanically.

Modify: Colour the plot by year or sector. Explain which assumption the visible pattern questions.

Regression residuals can also define a feature

Chapter 2 subtracted a sector mean with transform. Multiple regression can remove both sector averages and a numeric liquidity component from a supplied momentum score.

$\text{momentum}=\text{intercept}+\text{sector terms}+\beta\,\text{log volume}+\text{residual}$.

The residual is observed momentum minus its fitted component. This cross-sectional transformation uses one date; it does not forecast a future return.

One date’s inputs

Response	Controls
Momentum	Sector
Same stock rows	Log trading volume

Fit and subtract

Calculation	Output
Observed − fitted	Residual feature
Original row index	Retained

Inspect one cross-section before neutralising

Use nine stocks from the same monthly table. log_dvol_126d is the supplied log trading-volume measurement. These fixed sectors are classroom descriptions.

Fit the same OLS form to the feature

get_dummies turns sector categories into 0/1 columns. drop_first=True omits the reference sector; add_constant supplies the intercept. Here the response is the observed momentum feature.

Check what the residual transformation guarantees

OLS with an intercept and sector indicators gives approximately zero mean residual within each sector. It also makes residuals orthogonal to the included numeric control in this fitted sample.

These identities verify the fit. They do not demonstrate useful forecasts, causal effects or a sector-neutral portfolio. Repeating the transformation on another date requires a separate same-date fit.

Specification, redundancy and regularisation

Question: How do feature choices change bias and variance?

Result: Compare interactions, subsets, VIF and penalties within development data.

Builds on Coefficient uncertainty and residual checks.

Interactions and nonlinear terms change the question

\[E[Y\mid X]=\beta_0+\beta_1m+\beta_2v+\beta_3mv+\beta_4m^2.\]

The momentum slope becomes $\beta_1+\beta_3v+2\beta_4m$. With an interaction, its main coefficient is the slope at the other interacting variable’s zero value.

Feature construction uses information available at each row. Select transformations using training/validation, then freeze them before test evaluation.

Best subset is a search with a statistical cost

Five candidate regressors produce $2^5-1=31$ non-empty subsets. Searching them can overfit the validation set even though each OLS fit is simple.

The helper below iterates over model specifications, rather than processing data rows. It records every attempted subset. Adjusted R² is a useful diagnostic, but we select by the declared validation RMSE.

Multicollinearity inflates coefficient uncertainty

Regress feature j on the remaining regressors to obtain $R_j^2$:

\[VIF_j=\frac{1}{1-R_j^2}.\]

Near-collinearity makes separate coefficients unstable. Exact collinearity prevents unique identification. Prediction may remain stable within the observed feature region even while coefficients change sharply.

A VIF cutoff is a diagnostic convention, rather than a theorem that a feature should be deleted. Consider interpretation, redundancy, sample size and validation.

Penalties and scale

The ridge penalty discourages large coefficient magnitudes, reducing variance at the cost of bias. Its geometry depends on feature units. A coefficient measured per dollar and another measured per million dollars would receive very different effective penalties without scaling. Fit scale parameters on the training set and reuse them unchanged for validation or test. Refitting them on a later cross-section changes the transformation and needs its own declared rule.

Lasso can select a sparse representation, but with highly correlated features its chosen member may be unstable. A zero penalised coefficient is a property of the optimisation and penalty, rather than proof of a zero causal effect. Standard OLS coefficient p-values cannot simply be attached to a lasso-selected model and interpreted as though no selection occurred. Match the inferential claim to the full estimation procedure.

Ridge and lasso trade variance for bias

Ridge minimises $SSE+\lambda\sum_j\beta_j^2$. Lasso uses $SSE+\lambda\sum_j|\beta_j|$ and can set some coefficients to zero. The intercept is normally unpenalised.

Scaling matters because penalties act on coefficient magnitudes. Fit the scaler on training rows only. Choose penalty strength on validation data or time-aware folds.

Frozen selection and held-out evidence

Question: When can the final test be opened?

Result: A frozen specification and a clear distinction between interval types.

Builds on Specification, redundancy and regularisation.

Bias, variance and the role of the untouched test

For squared prediction loss, expected error can be decomposed into irreducible noise, squared bias and variance under a specified data-generating model.

More flexible fits often reduce training bias while increasing estimation variance. Regularisation and averaging can reduce variance. The trade-off depends on the sample and model, rather than a universal algorithm ranking.

%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 20}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
    A["Training<br/>fit parameters"] --> B["Validation<br/>compare rules"]
    B --> C["Freeze features<br/>settings and cutoff"]
    C --> D["Test once<br/>report limitations"]
    D --> E["New ideas need<br/>new evaluation"]

Self-check: selection does not establish causation

Self-check: Which is a defensible model-selection claim?

The subset with the smallest p-value must have the best future return
A higher training R² guarantees lower future RMSE
Validation can guide a fixed search, but repeated tuning consumes its independence and coefficients still need identification for causal claims
A large VIF proves every prediction is wrong

Feedback: Validation guides a declared search. Selection uncertainty, generalisation and causal identification remain distinct issues. #### A coefficient interval is not a future-return interval

Self-check: What does a coefficient confidence interval describe?

A band containing 95% of future stock returns
Uncertainty in a specified coefficient under the estimation and sampling assumptions
The probability that every strategy trade succeeds
The maximum possible drawdown

Feedback: Coefficient uncertainty concerns the estimated model parameter. A future-outcome prediction interval also includes outcome noise.

VIF diagnoses coefficient redundancy

Self-check: A large VIF most directly indicates what?

A proven causal relationship
Strong linear predictability of one regressor from the others
Guaranteed poor future portfolio performance
A calibrated classifier

Feedback: VIF reflects collinearity in X and potential inflation of coefficient variance. It does not independently determine prediction quality.

Nonlinear Models and Loss Classification

Part	Question
4.3.1 Trees, ensembles and feature use	How does a nonlinear learner make a prediction?
4.3.2 Event probabilities and classification scores	How good are the event probabilities?
4.3.3 Decision thresholds and held-out evaluation	Which probability should trigger a risk action?

Trees, ensembles and feature use

Question: How does a nonlinear learner make a prediction?

Result: Compare trees, forests and boosting on the same validation rows.

Begin with the definitions and assumptions; then trace the worked example.

Tree complexity and ensemble reasoning

A tree partitions feature space into regions. Within each region it predicts a constant mean or event fraction. A split can model an interaction because a later threshold may be applied only after an earlier condition holds. The resulting flexibility also creates high variance: small sample changes can move early splits and many downstream predictions. Depth limits and minimum leaf sizes constrain this variance, but their appropriate values depend on data and validation.

Bagging averages fitted predictors. If their errors were perfectly correlated, averaging would offer little variance reduction. Bootstrap sampling and random feature subsets encourage diversity while retaining useful structure. Boosting instead builds an additive function by successive loss-reducing updates. For general differentiable loss, the fitted pseudo-residual is the negative gradient, not necessarily the raw observed-minus-predicted difference. The squared-error case provides the simple residual intuition used here.

Rebuild the two targets for nonlinear learning

Regression targets a continuous return. Classification targets the fixed 5% loss event. Keep the split and candidate spine unchanged when comparing models.

A regression tree chooses a split by loss reduction

For a candidate split s, minimise the two child sums of squared errors:

\[\sum_{i\in L(s)}(y_i-\bar y_L)^2+\sum_{i\in R(s)}(y_i-\bar y_R)^2.\]

Each leaf predicts its training mean. Recursive splits capture thresholds and interactions automatically, but make piecewise-constant predictions.

flowchart TD
    A["Is volatility below the fitted threshold?"] -->|Yes| B["Is momentum below a second threshold?"]
    A -->|No| C["Leaf: mean training return"]
    B -->|Yes| D["Leaf: mean training return"]
    B -->|No| E["Leaf: mean training return"]

This diagram illustrates the mechanism. The fitted thresholds below come from training data.

Changing the fitted loss changes the prediction

For one insurance risk segment, should a tree predict the mean claim payout or the typical claim payout?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

Changing the fitted loss changes the prediction · Python

Payout is thousands of dollars per filed claim, in one fixed risk segment. No split is possible, so squared_error fits the training mean 3 and absolute_error the median 1. On the predeclared later fixture, the first wins MSE and the second MAE. Expected payout for budgeting and a typical claim are different targets. Both losses were declared before evaluation; these eight invented claims do not establish generalisation or total policy risk.

Fit a shallow tree and inspect its rule

Read each path as a sequence of conditional comparisons. The leaf’s predicted return comes from outcomes of training rows reaching that leaf.

Modify: Increase depth or reduce minimum leaf size. Compare training and validation error; do not select settings using test returns.

Classification trees use impurity or log loss

For class proportions $p_k$ in a node, Gini impurity is $1-\sum_k p_k^2$ and entropy is $-\sum_k p_k\log p_k$.

A split maximises the weighted reduction in impurity. A leaf estimates class probability from its training observations. A pure leaf can be a tiny, unreliable sample.

Classification impurity and regression SSE answer different loss functions. Neither automatically reflects the economic cost of a missed loss event.

Check: Why can a deep classifier attain perfect training accuracy yet give poor probability forecasts?

Forests average; boosting adds corrective learners

Bagging fits models on bootstrap samples and averages their predictions. A random forest also samples features at candidate splits, helping decorrelate trees. Averaging reduces variance most when errors are not perfectly correlated.

Gradient boosting adds a small learner in the direction that reduces the chosen loss. Under squared error, the negative gradient is the current residual.

\[F_m(x)=F_{m-1}(x)+\eta h_m(x).\]

Learning rate η, tree depth, leaf size and number of stages jointly control flexibility. Tune them on time-aware validation, rather than assuming one ensemble dominates.

Compare regression learners on the same validation rows

These settings are a small fixed demonstration, not an exhaustive optimisation. Compare with the OLS and constant baselines from §4.1. More models tried means more selection uncertainty.

Importance describes a model’s use of a feature

Impurity importance totals the training loss reductions credited to each feature. It can favour features with many possible split points.

Permutation importance measures how a declared evaluation score changes when a feature is shuffled. Correlated substitutes can obscure the contribution of either variable. Arbitrary shuffling can create implausible market observations or break time structure.

Neither measure is a causal effect. Report the model, sample, score and perturbation scheme. A stable predictive feature still needs an availability and implementation check.

Event probabilities and classification scores

Question: How good are the event probabilities?

Result: Explain log odds, confusion counts, discrimination and probability quality.

Builds on Trees, ensembles and feature use.

Probabilities, prevalence and calibration

A classifier’s score can rank cases well while its probabilities are too high or too low. A monotone transformation can preserve AUC while changing Brier score, log loss and every fixed probability threshold. Calibration asks whether events occur at the frequencies suggested by the probabilities. Ranking asks whether higher-risk cases tend to have higher scores. Thresholded decisions ask how errors translate into a declared cost. These are related but distinct evaluation questions.

Event prevalence changes the practical interpretation of a classifier. Accuracy can be high under a rule that always predicts the common class. Precision depends on prevalence as well as sensitivity and false-positive rate. A probability forecast trained in one regime may become miscalibrated when prevalence changes. Calibration plots need enough cases in each bin, and confidence statements must respect clustered or serially dependent observations.

A 30-day inactivity target is not a 60-day target

The next purchases occur after 7, 21, 45 and 80 days. Which customers count as inactive at each horizon?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

A 30-day inactivity target is not a 60-day target · Python

Y30 means no purchase in the next 30 days; Y60 means no purchase in the next 60. Neither means permanent churn. Only use complete follow-up windows or explicitly handle censoring. The same p30 is scored against different events here to expose a target mismatch; the two Brier scores are not evidence that one horizon is inherently easier or better. Churn risk does not identify who benefits from a retention offer.

Logistic regression models an event probability

\[p_i=P(Y_i\le-0.05\mid X_i),\qquad \log\frac{p_i}{1-p_i}=\beta_0+X_i^\top\beta,\qquad p_i=\frac1{1+e^{-(\beta_0+X_i^\top\beta)}}.\]

A coefficient changes log odds. $e^{\beta_j}$ is the conditional odds multiplier for a one-unit feature increase. Odds and probabilities differ.

Logit estimates coefficients by maximising Bernoulli likelihood, equivalently minimising log loss. Regularisation adds a penalty. A probability model requires calibration checks as well as ranking scores.

Fit the loss classifier with training-only scaling

The event rates are about 21.7% in training and 16.2% in validation. Validation AUC is about 0.544.

StandardScaler inside the pipeline learns its means and scales only during training. A numeric coefficient here measures a one-training-standard-deviation change; scaling dummy variables also changes their coefficient units.

Check: AUC above 0.5 in one validation sample does not establish a calibrated probability or a net economic benefit.

Confusion counts depend on a threshold

Predict an adverse event when $p\ge c$.

Actual / prediction	Predict event	Predict no event
Event	True positive (TP)	False negative (FN)
No event	False positive (FP)	True negative (TN)

Precision = TP/(TP+FP), recall = TP/(TP+FN), specificity = TN/(TN+FP). Accuracy can look high when events are rare.

Lowering c tends to increase both true positives and false positives. State how undefined ratios are handled when a denominator is zero.

Ranking and probability quality need separate scores

ROC AUC measures ranking across event/non-event pairs. ROC traces recall against false-positive rate as the threshold changes. Precision–recall curves are useful when positives are rare, and depend strongly on prevalence.

Brier score is $n^{-1}\sum_i(p_i-y_i)^2$. Log loss is $-n^{-1}\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)]$.

Calibration compares predicted probabilities with observed event frequencies. Small bins and dependent observations produce uncertain calibration estimates.

Use a training-prevalence baseline. AUC ignores threshold costs and probability calibration.

Decision thresholds and held-out evaluation

Question: Which probability should trigger a risk action?

Result: A validation-chosen threshold evaluated once on held-out outcomes.

Builds on Event probabilities and classification scores.

Deriving a cost threshold

Let p be the event probability. Predicting an event incurs expected false-positive cost (1−p)C_FP, while predicting no event incurs expected false-negative cost pC_FN, under the simplified binary-cost setup. Choose the event action when the first is smaller, giving p ≥ C_FP/(C_FP+C_FN). This derivation assumes calibrated probabilities and constant costs. Capacity constraints, variable exposure, transaction costs and heterogeneous loss severity change the decision problem.

The exercise’s costs of one and four are abstract. They illustrate a statistical decision rule, rather than estimate a trading strategy’s gains and losses. The empirical validation grid favours 0.65 here, despite the simple calibrated-model formula giving 0.20. That gap can reflect calibration error, finite-sample variation and the restricted grid. It is a reason to investigate, not a licence to claim the fitted optimum is a stable economic law.

The fixed classifier then performs poorly on the declared test. Report that finding without using the test to reverse-engineer a flattering threshold. Annual refitting in §4.6 is a separate predeclared procedure that incorporates newly available training outcomes. Its portfolio should be judged as a complete process, including the refitting schedule and risk policy, rather than borrowing the interpretation of one earlier fixed-model metric.

Interactive threshold and error costs

Four simulated cases: probabilities 0.10, 0.25, 0.55, 0.85; actual events 0, 1, 0, 1. False alarm costs 1, missed event costs 4.

Alarm threshold 0.50

Predict, then move: Which cases change at thresholds 0.20 and 0.80? Explain why the minimum toy cost does not identify the correct threshold for the real validation sample. #### Choose the decision threshold on validation

Suppose a false alarm costs 1 and a missed adverse event costs 4, in declared abstract decision-cost units. With calibrated probabilities and constant costs, the expected-cost threshold is $1/(1+4)=0.20$.

The model may be miscalibrated. This exercise also evaluates a fixed threshold grid on validation only.

The grid selects 0.65, costing 56 units versus 64 for never alarming and 83 for always alarming. This is a validation result, with tuning uncertainty. These abstract costs are not realised trading P&L.

Probability accuracy and fraud decisions are different

One transaction is fraudulent. Does the usual 0.5 cutoff minimise missed-fraud and investigation costs?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

Probability accuracy and fraud decisions are different · Python

Y is fraud confirmed within a specified 30-day outcome window. Fit probabilities with log loss or Brier; assess ranking with AUC; choose actions with declared costs and capacity. Under calibrated probabilities and constant costs, alert at p >= C_FP/(C_FP+C_FN). Otherwise validate the rule. This tiny validation comparison is not final test evidence.

Open the held-out test after freezing the rule

Test AUC is about 0.431, and Brier score about 0.2176. This weak result is useful evidence against assuming the fitted risk model generalises.

Inspecting this test to invent another feature or threshold would turn it into development data. A revised model needs a new evaluation protocol.

Self-check: probabilities and decisions

Self-check: Which claim is supported by a validation AUC of 0.544?

The model predicts a 54.4% return
Its default cutoff must be 0.5
The score showed modest event ranking in that sample, while calibration, costs and future performance need separate checks
The model should automatically block every trade

Feedback: AUC measures event ranking in a stated sample. It does not supply return, calibration or a decision threshold.

Repair the adverse-event definition

The event is a next-month simple return of −5% or worse. Fix the comparison without inventing labels for pending rows.

Log odds differ from probability

Self-check: A logit coefficient is log(2) per unit. What doubles for a one-unit increase, holding other inputs fixed?

The event probability at every starting value
The conditional odds
The expected next-month return
The classifier AUC

Feedback: Exponentiating a logit coefficient gives an odds multiplier. A probability change depends on the starting probability.

Ensemble methods change the estimation trade-off

Self-check: Under squared-error gradient boosting, the next learner fits what?

The current residual direction
The future test returns before they arrive
Only the largest feature values
A causal treatment effect

Feedback: The negative gradient of squared loss is proportional to observed minus current fitted value. General losses have different pseudo-residuals.

Causal Questions and Research Designs

Part	Question
4.4.1 Prediction, intervention and confounding	Would changing a predictor change the outcome?
4.4.2 Difference-in-differences	Can a comparison group identify a change?
4.4.3 Instrumental variables	Can external variation identify the effect?

Prediction, intervention and confounding

Question: Would changing a predictor change the outcome?

Result: Distinguish association from an intervention and identify confounding.

Begin with the definitions and assumptions; then trace the worked example.

Predicting spend does not identify offer profit

A randomised offer raises average revenue by 3 per customer. Should it be used if each offer costs 4?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

Predicting spend does not identify offer profit · Python

Invented randomised groups illustrate an estimated average revenue effect of 3, before uncertainty. With cost 4 the estimated net effect is -1; with cost 2 it is +1, assuming revenue is contribution before offer cost. High predicted spending alone does not identify incremental response. Observational adjustment, DiD and IV each need their own identification assumptions; evaluate incremental contribution and uncertainty.

Two questions that look alike

The earlier sections estimated a conditional return or an adverse-event probability. A different question asks what would happen if a trading firm required staff to adopt a risk dashboard. An observed association between adoption and later risk-control scores can support a predictive hypothesis, but it does not establish that requiring adoption would improve the score.

In this clearly simulated example, a hidden trait, risk discipline, affects both adoption and the outcome. Traders who adopt the dashboard might therefore score better even if the dashboard itself changes nothing. The numerical outcome is an abstract risk-control score, rather than realised money or an observed return. The simulation lets us compare a known causal effect with the naive fitted association.

The potential outcomes framework makes the gap precise. Each trader $i$ has two potential risk-control score outcomes, $Y_i(1)$ with the tool and $Y_i(0)$ without, and we observe exactly one. The causal effect $Y_i(1) - Y_i(0)$ is never observed for anyone — causal inference is a missing-data problem. What we can compute is the observed gap between users and non-users, which decomposes as

\[\underbrace{E[Y \mid T=1] - E[Y \mid T=0]}_{\text{observed gap}} = \underbrace{E[Y(1)-Y(0)\mid T=1]}_{\text{causal effect on users}} + \underbrace{E[Y(0)\mid T=1] - E[Y(0)\mid T=0]}_{\text{selection bias}}.\]

The second term asks: would the users have achieved higher risk-control scores than the non-users even without the tool? If the people who install a risk dashboard were already disciplined about money, the selection term is large and positive and the observed score gap is mostly selection. Randomised assignment removes systematic selection in expectation under the design — that is why an A/B test is the gold standard — but most business data are observational, and the rest of this section is about what to do then.

A great predictor that is a useless lever

The cleanest way to see the problem is to build a world in which we know the answer. A hidden trait $U$ — risk discipline — drives both tool adoption $T$ and the risk-control score $Y$. By construction, the tool itself has zero effect on risk-control score.

36 % of traders use the tool, and the correlation between usage and risk-control score is 0.48. The naive regression of $Y$ on $T$ gives a coefficient of 2.17 with a t-statistic of 24 and an R² of 0.23. This is a strong in-sample association in the simulated population, with a large test statistic. It still requires a separate evaluation before any claim about future prediction. As a policy, pushing the tool would move risk-control score by exactly 0, because the world was built that way. The 2.17 is entirely selection: disciplined traders adopt the tool, and disciplined traders have higher risk-control scores. The t-statistic of 24 is not a lie — the association is real and would replicate in any sample — it just answers a different question from the one the manager asked.

The graph, not the coefficient

Before interpreting any coefficient as an effect, draw the arrows. Here they are $T \leftarrow U \rightarrow Y$: $U$ is a confounder, a common cause of treatment and outcome, and the path through it is a “backdoor” that carries association without causation. A predictive model does not care which arrows exist; a causal claim depends on nothing else driving both $x$ and $y$.

Regression adjustment: control for the confounder

If the confounder is measured, the fix is to put it in the regression. Conditional on $U$, the comparison is between tool users and non-users with the same discipline, the backdoor path is closed, and the coefficient on $T$ estimates the causal effect.

Add $U$ as a second regressor. Will the coefficient on $T$ stay near 2.17, collapse toward 0, or flip sign?

The adjusted coefficient on $T$ is 0.066 — close to zero in this simulation — and the coefficient on $U$ is 1.94, close to the true 2.0. In a second version of the world where the tool genuinely adds 0.5, the naive regression says 2.67 and the adjusted one says 0.45: the adjustment recovers the effect both when it is absent and when it is present.

Two cautions. First, this works only because $U$ was measured; in the real platform “risk discipline” is not a column. Second, the rule is include confounders, not include everything. A mediator — a variable on the path from $T$ to $Y$, such as “risk alerts received” — must be left out, because conditioning on it removes the effect you want to measure. A collider — a variable caused by both $T$ and $Y$, such as “selected for a testimonial” — must also be left out, because conditioning on it opens a spurious path (among professional basketball players height and skill are negatively correlated: being short and skilled is the only way a short player makes the league). Which variables to include is decided by the graph, not by their t-statistics.

When the confounder is not in your data

        U  (unobserved)
       / \
      v   v
      T -> Y

Regression adjustment cannot close a backdoor through a variable you do not have, and no amount of data on $T$ and $Y$ alone can recover the effect — more rows only give a more precise estimate of the wrong number. If you can randomise, do: an A/B test that assigns the tool by lottery makes $T$ independent of $U$, and a difference of means is then a causal estimate with a standard error from a two-sample t-test. When you cannot randomise, two observational designs replace the missing control with a structure. Difference-in-differences uses a control group that shares the treated group’s time trend. Instrumental variables use a source of variation in $T$ that has nothing to do with $U$. Each replaces the untestable “no unmeasured confounder” with a different, and sometimes more defensible, assumption.

Difference-in-differences

Question: Can a comparison group identify a change?

Result: Calculate DiD and state the parallel-trends assumption.

Builds on Prediction, intervention and confounding.

Difference-in-differences: the gap between gaps

Suppose the tool is rolled out to one region (the treated group) between period 0 and period 1, and not to another (the control). Two naive comparisons are available and both are wrong: treated versus control after the rollout picks up any pre-existing level difference, and the treated region before and after picks up whatever happened to everyone — a recession, a rate change, seasonality. DiD subtracts one from the other:

\[\hat\tau_{\text{DiD}} = (\bar Y^{\text{tr}}_{1} - \bar Y^{\text{tr}}_{0}) - (\bar Y^{\text{ctl}}_{1} - \bar Y^{\text{ctl}}_{0}).\]

The control group’s change stands in for what the treated group would have done without the policy. That works if, absent treatment, both groups would have moved in parallel — the parallel trends assumption. Level differences are fine, because they cancel in the first differences; different trends are fatal, because the control’s change is then not the treated group’s counterfactual. The simulation gives the treated group a level 1.0 higher, gives everyone a common drift of 0.5, and adds a policy effect of 0.8 to the treated group in period 1.

The four cell means are 5.013 and 5.466 for the control group, 6.001 and 7.315 for the treated. The naive post-period gap is 1.849 — the true 0.8 plus the 1.0 level difference. The naive before-after change in the treated group is 1.314 — the true 0.8 plus the 0.5 common drift. The DiD is $1.314 - 0.453 = $ 0.861, within a standard error of 0.8.

The regression version, $y = \alpha + \beta\,\text{group} + \gamma\,\text{period} + \delta\,(\text{group} \times \text{period})$, returns exactly the same four numbers as coefficients — $\hat\alpha = 5.013$ is the control’s baseline, $\hat\beta = 0.988$ the level difference, $\hat\gamma = 0.453$ the common drift, and $= $ 0.861 the DiD — now with a standard error of 0.051, so that the effect is 17 standard errors from zero and 1.2 from the truth. The interaction term is the whole design: the effect that exists only for the treated group and only after treatment. Counterfactual parallel trends cannot be verified directly. Pre-treatment patterns can challenge the assumption but cannot prove it. The simulation uses independent observations in each group-period cell; real repeated-unit designs need standard errors matched to their assignment and dependence structure.

Instrumental variables

Question: Can external variation identify the effect?

Result: Explain relevance, exclusion and the weakness of a weak instrument.

Builds on Difference-in-differences.

Instrumental variables: borrow a lottery

An instrument $Z$ is a variable that moves $T$ but touches $Y$ only through $T$. Three conditions must hold. Relevance: $Z$ shifts $T$ — testable from the first-stage regression. Independence: $Z$ is unrelated to the unobserved $U$ — usually argued from the design (a lottery, a weather shock, an arbitrary administrative rule). Exclusion: $Z$ has no path to $Y$ except through $T$ — never verifiable from data, defended on substantive grounds, and where most instrumental-variable arguments are attacked. Classic instruments are distance to the nearest college for schooling (Card 1995), the draft-lottery number for military service (Angrist 1990), and rainfall for agricultural income. On the simulated trading platform, a randomised in-app prompt would do: it moves adoption, it was assigned by lottery, and it has no reason to affect risk-control score except through the tool.

Two-stage least squares turns the instrument into an estimate. Stage 1 regresses $T$ on $Z$ and keeps the fitted values $\hat T$ — the part of $T$ that the instrument explains, which by independence is free of $U$. Stage 2 regresses $Y$ on $\hat T$. With a single instrument the whole procedure collapses to a ratio, $\hat\beta_{\text{IV}} = \text{cov}(Z, Y)/\text{cov}(Z, T)$: how much $Y$ moves per unit of $Z$, divided by how much $T$ moves per unit of $Z$. The cell does both stages by hand with np.linalg.lstsq, then repeats the exercise with an instrument that barely moves $T$.

The true effect of $T$ on $Y$ is 1.0. OLS says 2.013 — twice the truth — because $U$ pushes $T$ and $Y$ in the same direction and OLS attributes all of that co-movement to $T$. Two-stage least squares lands on 1.027, and the covariance ratio gives the identical 1.027, as it must with one instrument. The instrument has done what regression adjustment could not: it recovered the effect without ever observing $U$.

The second half shows the price. When the instrument’s coefficient in the first stage is cut from 0.5 to 0.05, the first-stage F-statistic falls from 637 to 7.0, and the 2SLS estimate drifts to 0.72 in this particular random draw. A weak instrument explains so little of $T$ that $\hat T$ is mostly noise, and the second stage is regressing $Y$ on noise; the estimate is biased toward the OLS value in finite samples and its confidence interval is unreliable. A first-stage F above 10 is a context-dependent heuristic rather than sufficient evidence for valid identification. The hand calculation reports point estimates. Ordinary second-stage OLS standard errors do not provide valid IV inference because they ignore the first-stage estimation structure.

Design	Replaces the missing control with	Fails when
Regression adjustment	the measured confounder	a confounder is unmeasured
Difference-in-differences	a control group’s time trend	trends are not parallel
Instrumental variables	an exogenous shock to $T$	$Z$ is weak or has its own path to $Y$

Each row of the table trades one untestable assumption for another, and none of them is free. The discipline that matters is not choosing the cleverest design but stating, in the same sentence as the estimate, which assumption it rests on: “the tool raises risk-control score by 0.86, assuming the two regions would have moved in parallel”. A causal number without its assumption is a prediction wearing a disguise.

Mistakes library: a useful predictor becomes an unsupported policy

In the simulated dashboard case, the naive adoption coefficient is 2.17 even though the treatment effect is zero. Requiring adoption on the basis of that coefficient would mistake selection for an intervention effect. Adding a measured confounder repairs this simulation because its causal structure is known; adding more predictors indiscriminately is not a general solution. State the graph and identification assumptions before recommending a policy.

The hidden trait $U$ (discipline) drives both adoption and risk-control score: $T \leftarrow U \rightarrow Y$. The association is real and replicable — it is a fine predictor — but it is entirely selection bias, $E[Y(0)\mid T=1] - E[Y(0)\mid T=0]$. Adding $U$ to the regression closes the backdoor and the coefficient collapses to 0.066. A t-statistic measures the strength of an association, never its direction of causation.

DiD: absent treatment, the treated and control groups would have followed parallel trends — not directly verifiable as a counterfactual; pre-period evidence can challenge it. IV: relevance (assess first-stage strength; F > 10 is only a heuristic), independence of $Z$ from unobserved confounders (argued from design), and exclusion — $Z$ affects $Y$ only through $T$ — which can never be verified from data alone.

Cross-sectional Attention Features

Part	Question
4.5.1 Snapshots and comparable feature scales	Which stocks and features belong in the same snapshot?
4.5.2 Attention weights and peer context	How are peer observations combined?
4.5.3 Walk-forward feature comparison	Does contextual information improve a fixed baseline?

Snapshots and comparable feature scales

Question: Which stocks and features belong in the same snapshot?

Result: A dated matrix with explicit universe and scaling limitations.

Begin with the definitions and assumptions; then trace the worked example.

Attention can describe comparable stores

Can a store use peer information without looking at those stores' future sales?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

Attention can describe comparable stores · Python

Rows and columns identify stores in the same snapshot. Affinity is exp(-distance/temperature), normalised across peers; these fixed weights are an attention-style illustration, not a trained Transformer. Values must be known before the forecast. A smoother context is a candidate feature; compare it with an unchanged baseline on later data under the chosen sales loss.

One month, one matrix

Every month a systematic equity fund sees the same object. At month-end $t$ there is a matrix $X_t \in \mathbb{R}^{N \times K}$ — $N$ stocks by $K$ characteristics — and a vector $y_t \in \mathbb{R}^N$ of next month’s returns, the target. The problem is cross-sectional: rank the $N$ stocks within the month so that the top outperforms the bottom. The single-series machinery of Section 4.3 is replaced by a stack of monthly snapshots, and the modelling questions change with it: how to standardise a snapshot without using other snapshots, what “peer group” a stock should be judged against, and how to score a ranking rather than a point forecast.

25 335 rows: 119 months × 213 stocks in 11 sectors, from January 2015 to November 2024. Five price-based characteristics are the features: last month’s return ret_1_0, the 12-month return skipping the most recent month ret_12_1 (the standard momentum signal), 21-day realised volatility, log dollar volume over 126 days, and the price as a fraction of its 252-day high. The target ret_next at month $t$ is the return over month $t+1$ — the check on AAPL confirms it is exactly ret_1_0 at the next row, 0.100776 in both places. The descriptives show why raw characteristics cannot go straight into a model: log_dvol_126d averages 19.5 while prc_highprc_252d is a fraction near 0.9, and ret_12_1 runs from −0.93 to +9.34 (a stock that rose tenfold in a year).

Whose 213 stocks are these?

The universe is today’s large caps, tracked back to 2015. Nothing that was large in 2015 and has since failed, shrunk or been delisted is in the file. That is survivorship bias: membership uses information unavailable in 2015 and omits failed or delisted firms. The direction of its effect on a long-short spread is not guaranteed, because both legs change. The production rule is to build the universe as of each month from a point-in-time index membership file. This file is fine for learning the mechanics — never quote its Sharpe ratio as evidence.

Standardise per snapshot, never across time

A z-score of rvol_21d computed over the whole 2015–2024 file uses the mean and standard deviation of nine years that, for the 2015 row, had not yet happened. A model run live in January 2015 would have seen a different number, so the backtest no longer reproduces what the model could have known: a look-ahead hidden inside a preprocessing step. The transform is monotone, which is why the bug is easy to miss — the ordering within a month is unchanged — but the scale now depends on the future. A refitted tree can be invariant to an affine transformation; this does not make the availability contract valid, and distance-based models and regularised estimators can change.

The fix is to standardise within each month, using only the cross-section observed at $t$: subtract the month’s median and divide by the month’s standard deviation. The median rather than the mean, because a single stock that rose tenfold should not shift the centre; and clip at ±3, so that one extreme month cannot dominate a tree split. In production the same transform is applied within each month × sector, so that a stock is judged against the peers it is actually compared with on the day — the _s columns below.

groupby(...).transform returns a frame aligned to the original rows, so the subtraction and division are row-by-row against each row’s own month (or month × sector). The largest sector median after standardising is exactly 0, as it must be; the clipped columns have standard deviation 0.91 (market-relative) and 0.97 (sector-relative), slightly under 1 because of the clipping, and a mean of 0.07 because the distributions are right-skewed and the median sits below the mean. Ten features so far: five market-relative, five sector-relative.

Attention weights and peer context

Question: How are peer observations combined?

Result: Trace similarity, temperature and context into a feature vector.

Builds on Snapshots and comparable feature scales.

Why a sector median is a crude peer group

“Sector” is a label assigned by an index provider. It is stable, cheap and often wrong for the purpose at hand. In October 2022 NVIDIA’s behavioural peers — high dollar volume, high volatility, far below the 52-week high after a year of falling — included AMD, which shares its sector, but also Netflix, Alibaba and Tesla, none of which is “Technology”. A sector-relative z-score compares NVIDIA with Apple, Microsoft and Oracle, which in that month looked nothing like it.

Attention builds the peer group from the data instead. The construction is the one at the core of the transformer, and it is worth understanding from first principles because it is far simpler than its reputation. Start with the matrix $X$ of $N$ stocks by $K$ standardised characteristics. Three linear maps produce three views of each stock: a query $q_i = W_q^\top x_i$, “what am I looking for in a peer?”; a reference vector $k_j = W_k^\top x_j$, “what do I offer as a peer?”; and a value $v_j = W_v^\top x_j$, “what information do I contribute once chosen?”. The score of stock $j$ as a peer of stock $i$ is the inner product $q_i \cdot k_j$, scaled by $\sqrt{d}$ (the dimension of the reference vectors, so that the scale of the scores does not grow with $d$) and by a temperature $\tau$. The scores in row $i$ are then passed through a softmax,

\[A_{ij} = \frac{\exp(q_i \cdot k_j / \sqrt{d}\,\tau)}{\sum_{l} \exp(q_i \cdot k_l / \sqrt{d}\,\tau)},\]

which turns them into positive weights summing to one — a probability distribution over peers. Finally the context of stock $i$ is the weighted average of the values, $c_i = \sum_j A_{ij} v_j$. In matrix form, with $Q = XW_q$, $K = XW_k$, $V = XW_v$:

\[A = \text{softmax}_{\text{rows}}\!\left(\frac{QK^\top}{\sqrt{d}\,\tau}\right), \qquad C = AV.\]

Now set $W_q = W_k = W_v = I$. The score becomes $x_i \cdot x_j / \sqrt{K}$ and the weight becomes proportional to $\exp(x_i \cdot x_j / \sqrt{K})$: a similarity kernel in characteristic space. Stocks that look like $i$ — the same sign and size of momentum, volatility, volume, drawdown — get high weight; stocks that look unlike it get weight near zero; and the sector label never enters. The context $c_i$ is then the characteristic vector of $i$’s data-driven peer group, and the deviation $x_i - c_i$ says how $i$ differs from that group. That is precisely what a sector-relative z-score was trying to be, with the peer group chosen by the data instead of by an index committee.

Twelve lines. The only numerical care is the subtraction of the row maximum before exponentiating, which prevents overflow and does not change the softmax (a constant in the exponent cancels between numerator and denominator). NVIDIA’s attention row in October 2022 gives AMD 0.353, NVIDIA itself 0.263, Netflix 0.154, Alibaba 0.077, Tesla 0.055, Amazon 0.028 — six stocks carry more than 1 % of the weight, and they come from three sectors. The context vector is the peer-weighted characteristic vector: NVIDIA’s own 12-1 momentum is −1.49 standard deviations against −1.38 for its peers, its dollar volume 3.0 (clipped) against 2.52, its distance from the 52-week high −2.67 against −2.59. In that month NVIDIA largely was its peer group — a cluster of large, liquid, volatile stocks in deep drawdown — and no sector median would have seen it.

Temperature: the dial between two crude peer groups

The temperature $\tau$ divides the scores before the softmax, and it controls how concentrated the weights are. As $\tau \to 0$ the scores are magnified, the softmax approaches a hard maximum, and all the weight goes to the single most similar stock. As $\tau \to \infty$ every score goes to 0, every weight goes to $1/N$, and the context collapses to the cross-sectional mean.

With identity weights and $\tau = 10^6$, what does NVIDIA’s context vector become? And at $\tau = 0.05$, does NVIDIA attend mostly to itself?

At $\tau = 10^6$ the context is (0.02, 0.10, 0.19, 0.00, −0.14) — exactly the cross-sectional mean of $X$, which median-centred, clipped standardisation need not remove. This context is identical for every stock within a date, so it provides no additional within-date discrimination. At $\tau = 0.05$ the weight on AMD is 0.997 and NVIDIA’s self-weight is 0.003: NVIDIA is not even its own nearest point, because the score is an inner product rather than a distance, and AMD’s vector, pointing the same way with a larger norm, scores higher. At $\tau = 5$ the weights are nearly uniform (self 0.025, largest other 0.027). $\tau = 1$ sits in between — a handful of behavioural peers — and is the setting used below; it is a hyper-parameter, and the exercise at the end of the section is about what happens when it moves.

What Kelly, Kuznetsov, Malamud and Xu actually train

The idea of putting attention over the stock cross-section rather than over words is due to Kelly, Kuznetsov, Malamud and Xu, “Artificial Intelligence Asset Pricing Models” (2025). Their architecture differs from the toy above in three ways. Each layer uses linear attention, $A = XW_qW_k^\top X^\top$ with no softmax, followed by a feed-forward block and a residual connection, $X^{(\ell+1)} = X^{(\ell)} + \text{FFN}(AX^{(\ell)}W_v)$, stacked $L$ times. The final layer maps each stock’s row to a portfolio weight, and every $W$ is trained end-to-end to maximise the Sharpe ratio of that portfolio — not to minimise a squared error. And they train on thousands of stocks over six decades. Attention is what lets a stock’s weight depend on the whole cross-section: the peer group is learned, for the objective the fund actually cares about.

What we do instead

The weights stay fixed at $W = I$ (a seeded random $W$ works the same way and is the “random features” view of the same construction). The attention output is used only as feature engineering: for every stock-month, the context $C$ — peer-weighted characteristics — and the deviation $X - C$, “how I differ from my attention-weighted peers”. The learner on top is the boosting machine from Section 4.3, fitted to next month’s return by squared error. Nothing is trained on a Sharpe objective, and the comparison is between three feature sets under one protocol, not between architectures.

attention_features runs one attention pass per month — only month-$t$ rows enter month $t$’s attention matrix, so there is no look-ahead — and returns ten columns per stock-month, aligned to the original frame by index rather than by position (the groupby does not preserve row order, and assigning by position would scramble stocks across months). Fifteen features now: five market-relative characteristics, five peer contexts, five deviations. NVIDIA’s October 2022 row reads: momentum −1.49 against a peer context of −1.38, a deviation of −0.10; price-to-high −2.67 against −2.59, a deviation of −0.08. With a self-weight of 0.26 and AMD at 0.35, NVIDIA’s deviations from its peer group are small, which is the correct description of that month.

The rest of the cell sets up the protocol. The test period starts in January 2019, giving 71 test months; the model is refitted every 12 months on all rows dated strictly before the block and predicts the next 12 — 6 refits, the first on 10 212 rows (monthly refitting would require 71 fits; its results are not evaluated here). The learner is HistGradientBoostingRegressor with depth-2 trees, learning rate 0.05, 200 rounds and a minimum leaf of 50 — Section 4.3’s recipe in its faster, histogram-based form.

One subtlety deserves a sentence. The row dated 2018-12-31 has ret_next equal to January 2019’s return, which is realised on 31 January 2019 — the same instant at which the 2019-01-31 features are observed. So when we predict the 2019-01-31 row, the 2018-12-31 target is already known, and training on all rows dated strictly before the test block is legal. A two-month-ahead target would need a one-month gap between the last training row and the first test row; a one-month-ahead target does not.

Two scores are computed per month. The rank IC is the Spearman correlation between the model’s prediction and the realised return across the 213 stocks — the natural score for a ranking problem, immune to the scale of the predictions. The top-20 minus bottom-20 spread is the equal-weight return of the 20 highest-ranked stocks minus the 20 lowest — the return of the simplest long-short portfolio the ranking could produce, before costs.

Walk-forward feature comparison

Question: Does contextual information improve a fixed baseline?

Result: A common-protocol comparison with uncertainty and sample limits.

Builds on Attention weights and peer context.

Three feature sets, one protocol

Form an expectation before running. The raw five features are the smallest set and the hardest to overfit; the sector-relative set is the within-date transformation rule; the attention set has data-driven peers. Which will win — and will the difference be detectable?

Mean rank IC: −0.015 (t = −1.04) for the raw features, −0.014 (t = −0.99) with sector features, −0.010 (t = −0.67) with attention. The approximate unadjusted tests do not reject zero, which does not establish zero effects — the standard error of a mean IC over 71 months is about 0.015 — and all three are, if anything, slightly negative. The long-short spread tells a slightly different story: +7.7 %, +9.3 % and +2.8 % a year, with Sharpe ratios of 0.47, 0.62 and 0.17. The extremes of the ranking do a little better than the middle even though the ranking as a whole does not, which is common: the tails of a prediction are where the model is most confident and the signal, if any, is most concentrated.

Attention features did not help here. The attention set has the least negative IC and the lowest Sharpe; the ranking between the sets flips with the score you read; the reported standard errors describe individual mean ICs, rather than paired differences. To compare models formally, analyse paired monthly differences with dependence-aware uncertainty. That is the honest result, and the honest result is the deliverable. Five price-based characteristics on 213 large caps over 71 months is a small, noisy problem — KKMX have thousands of stocks and six decades — and a feature that needs that much data to show its value cannot be expected to show it here. What the protocol delivers is the ability to tell: had the split been shuffled, or the standardisation done over the full sample, the attention set could easily have “won”, and the win would have been an artefact.

Is a Sharpe ratio of 0.6 evidence?

The plot shows three cumulative spreads drifting upward with large swings, and the temptation is to read the 0.62 Sharpe as a modest real edge. The arithmetic says otherwise. The t-statistic of a monthly spread with annualised Sharpe $S$ observed for $T$ months is $S\sqrt{T/12}$: with $T = 71$, the sector set’s 0.62 gives t = 1.51, the raw set’s 0.47 gives 1.15, attention’s 0.17 gives 0.40. To reach t = 2 the sector set would need 124 months of the same performance, the raw set 18 years. A Sharpe ratio is a point estimate with standard error roughly $\sqrt{12/T} \approx 0.41$ here — wider than the differences between the three lines — and the file also has survivorship bias. These calculations assume independent, stable monthly observations; serial dependence and model selection change uncertainty.

Changing the peer group

The peer-group definition has two dials: the temperature, and which characteristics form the reference vectors. The cell rebuilds the attention features using only momentum and volatility as reference vectors — so that “similar” means “similar momentum and volatility”, ignoring volume and drawdown — and reruns the walk-forward on the attention set.

Momentum-and-volatility reference vectors move the mean IC to −0.025 (t = −1.64), from −0.010. Try the other settings and the pattern is the same: $\tau = 3$ leaves the IC at −0.010, $\tau = 10$ gives −0.008, and $\tau = 0.3$ gives −0.041 with t = −2.60 — the one “significant” number, and it appears on the fourth try. The sign of the change flips with the setting, and a t-statistic of 2.6 found after four searches is multiple testing, not a signal: with four independent tries, the chance that at least one exceeds |t| = 2 by luck is about 18 %. In KKMX the peer group is learned on a Sharpe objective with thousands of stocks and six decades; here it is a fixed kernel on 213 survivors, and the data cannot say which kernel is better. The chapter ends where it began: the protocol is the product, and the number it reports is only as good as the discipline behind it.

$A_{ij} \propto \exp(x_i \cdot x_j / \sqrt{K}\,\tau)$ — a similarity kernel in characteristic space, independent of sector labels. As $\tau \to 0$ all weight goes to the single most similar stock (at $\tau = 0.05$ NVIDIA gave AMD 0.997); as $\tau \to \infty$ every weight is $1/N$ and the context is the cross-sectional mean, which need not be zero after median centring and clipping. It is identical across stocks within the date.

The 2015 z-score would use the 2015–2024 mean and standard deviation, numbers a live model in 2015 could not have seen; the ordering within a month is unchanged but the scale depends on the future. Some refitted tree predictions are invariant to affine scaling, while other learners are sensitive; the transform still violates the declared availability clock. Standardise within each month (median / std across the stocks observed at $t$, clipped at ±3), and within month × sector when the peer group is the sector.

Walk-forward Decisions and Portfolio Accounts

Part	Question
4.6.1 Frozen forecasts and stateful refits	When is each forecast formed and which labels are mature?
4.6.2 Risk rules, holdings and cash	What happens to uninvested capital and trading costs?
4.6.3 Comparison and the decision memo	What evidence supports the proposed policy?

Frozen forecasts and stateful refits

Question: When is each forecast formed and which labels are mature?

Result: Forecasts with stock and date columns from an availability-aware refitting procedure.

Begin with the definitions and assumptions; then trace the worked example.

Wait for the defined outcome before refitting

At 10 February, can a model train on a January customer whose full 30-day outcome is still unknown?

Compare

Invented teaching data; comparisons are illustrations, not measured deployment gains.

Wait for the defined outcome before refitting · Python

The displayed table is a later export. On 10 February only the first outcome was available; even the second row’s eventual zero was not yet known. Keep new predictions and pending outcomes. Predeclare the target, fitting loss, validation metric and action rule; then evaluate later realised contribution, service levels and costs. A corrected objective may improve alignment, but generalisation must be measured.

Why accounting is part of statistical evaluation

An allocation table is a policy, while an account is its realised financial path under stated execution assumptions. Holdings drift as asset prices move. Rebalancing to the next target trades against those drifted values. A turnover calculation that compares successive target weights can therefore differ from actual traded notional. Initial purchases, cash balances and fees also affect NAV. Reconciliation detects accounting mistakes before a performance statistic is interpreted.

The teaching account allocates 99% before fees and leaves 1% as a buffer. Blocked stock allocations remain cash. It assumes zero cash interest, no leverage, no borrow and a fixed ten-basis-point proportional fee on both purchases and sales. These assumptions define the account being measured. Real implementation also needs capacity, spreads, market impact, corporate actions, delistings and execution timing. This sample does not establish those conditions.

A paired comparison with a changing exposure

The baseline and risk-filtered account use the same dates and candidate universe, making their monthly return difference a paired statistic. The filter can change exposure and turnover as well as which stocks are held. A lower drawdown could arise partly from holding more cash. Report that exposure change alongside return and fees, and consider an exposure-matched comparison if the research question asks about selection skill rather than simply reduced investment.

Chapter 3 explains why nine stocks per month do not create nine independent portfolio return observations. There are only 34 realised periods in this account. A block bootstrap or other suitable time-aware interval can describe sampling uncertainty under its assumptions, but more resamples cannot create new stress episodes. A reconciled path and an attractive ending NAV cannot establish production readiness, causal risk reduction or approval to trade.

A forecast becomes a portfolio only through an explicit rule

%%{init: {"flowchart": {"nodeSpacing": 24, "rankSpacing": 20}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
    A["Return<br/>forecast"] --> C["Rank stocks<br/>select top three"]
    B["Loss<br/>probability"] --> D["Apply frozen<br/>risk threshold"]
    C --> D
    D --> E["Target weights<br/>retain cash"]
    E --> F["Trade from<br/>drifted holdings"]
    F --> G["Costs, realised<br/>P&L and NAV"]

The return model ranks candidates. The classifier can reduce exposure under a declared policy. Evaluate each statistical model separately, then evaluate the complete net account. Weak classification evidence remains a limitation even if one portfolio path looks good.

Rebuild an availability-aware walk-forward sample

For each calendar year, refit using labels available strictly before 1 January. Predict every month in that year using its available features. This conservative annual protocol differs from refitting at each month-end.

Ridge alpha 100 (selected by the fixed validation grid), logit C 1, the 5% event definition and risk threshold 0.65 are frozen settings. The threshold grid was studied using 2021 validation. Do not retune them using the subsequent portfolio outcomes.

A function isolates the stateful refitting logic

There are 315 predictions, 306 evaluable outcomes and 9 pending predictions. Preprocessing is refitted within each training sample. Stock/date columns beside predictions survive concatenation.

The loop represents three successive fitting clocks. It does not mix future outcomes into an earlier fit. Exact execution, historical membership and vendor timestamps remain unverified classroom assumptions.

Risk rules, holdings and cash

Question: What happens to uninvested capital and trading costs?

Result: A holdings, cash and fee ledger that reconciles through time.

Builds on Frozen forecasts and stateful refits.

Risk filtering retains cash instead of silently releveraging

At the frozen 0.65 threshold, this sample blocks none of the selected positions, so the baseline and filtered accounts are identical. This rule supplies no realised risk reduction here. An allowed selected stock keeps weight 0.33. A blocked stock receives zero, and its allocation stays in cash. Rescaling the survivors to 99% would define a different risk policy.

The row order provides a deterministic tie rule in this exercise. A production specification must state tie-breaking, capacity, borrow and concentration constraints.

Accounting follows holdings, cash and fees through time

This function carries holdings and cash across periods, so a short stateful loop is appropriate. Trades start from drifted holdings, rather than last month’s target weights. Cash earns zero in this exercise.

Both accounts contain 34 realised monthly holding periods. Initial purchases incur fees. NAV is measured after fees and realised simple-return P&L. The identity NAV change = gross P&L − fees must hold at every row.

Comparison and the decision memo

Question: What evidence supports the proposed policy?

Result: A paired comparison that reports exposure, uncertainty and pending outcomes.

Builds on Risk rules, holdings and cash.

Compare net outcomes with their exposure and uncertainty

Report the actual values produced, exposure differences, turnover and fee assumptions. Apply Chapter 3’s paired, dependence-aware uncertainty to the monthly return difference.

Thirty-four months and nine surviving stocks cannot establish a reliable risk reduction. A net account can reconcile perfectly while its model generalises poorly.

Self-check: cash is part of the policy

Self-check: One of three 0.33-weight stocks is blocked by the risk rule. What follows from this policy?

The other stocks automatically rise to 0.495 each
The missing allocation stays in cash, and exposure falls from 0.99 to 0.66 before fees
The blocked stock’s return becomes zero in the source file
No transaction can occur

Feedback: The declared filter keeps the original weights on allowed names and sends blocked allocations to cash. Renormalising creates a different policy.

The decision memo links statistical evidence to action

Report target and horizon; candidate/label counts; feature availability; training/validation/test clocks; fitted models; baseline metrics; frozen decision costs and threshold; exposure, net fees and drawdown; uncertainty and limitations.

The fixed classifier’s test AUC of 0.431 does not support claiming a dependable loss detector. The attention extension does not establish that a transformer learns a profitable strategy. Keep negative findings visible.

Your investigation: Change one declared assumption on a development sample, record the attempt, explain the changed predictions and account, and propose fresh evidence for evaluation. Do not rename an examined test “untouched”.

Pending predictions stay outside realised accounting

Self-check: The last nine candidates have pending outcomes. What is appropriate?

Delete them before predicting
Predict them if features are ready, but omit their unobserved holding periods from realised evaluation
Fill future returns with the training mean
Use their label-status flags as return predictors

Feedback: Prediction eligibility and outcome availability are separate. Realised accounting needs actual mature outcomes.

Reconciliation is necessary evidence

Self-check: If NAV change equals P&L minus fees at every date, what has been established?

The strategy is profitable out of sample
The specified accounting identity holds for the supplied inputs
All source records were known historically
The model has a causal effect

Feedback: Reconciliation checks accounting consistency. Source availability, generalisation, causality and execution require separate evidence.

Chapter Wrap-up

You can define a continuous outcome and a binary adverse event, build an explicit feature matrix, fit and interpret multiple regression, evaluate assumptions and coefficient uncertainty, compare specifications using chronological validation, and distinguish ranking, calibration and threshold decisions. Causal designs require identification assumptions beyond predictive association. Fixed attention provides a transparent feature-engineering extension rather than a trained transformer or an economic guarantee.

The final walk-forward account preserves pending prediction candidates and evaluates only mature outcomes. It carries holdings and cash through time and reconciles NAV after fees. Keep the weak fixed-classifier test result, small survivor universe, static sectors and assumed availability clocks visible in the recommendation. Chapter 5 develops Bayesian uncertainty and Chapter 6 adds time-series structure.

Method references

statsmodels OLS and robust covariance.
scikit-learn linear models, trees, ensembles and classification metrics.

Chapter Introduction

How to use the numbered learning path

Table of Contents

Targets, Regression and OLS

Targets, rows and information clocks

Why the target definition comes before the model

Define the delivery outcome before choosing a model

Define the delivery outcome before choosing a model · Python

One table, two statistical questions

The modelling path preserves the data contract

Load the case and freeze the three clocks

Read the feature matrix before fitting

Fitting and interpreting OLS

A different loss can require a different stock level

A different loss can require a different stock level · Python

Multiple regression separates conditional associations

Population errors, fitted residuals and influence

OLS chooses coefficients by squared residuals

Fit OLS and compare training with validation

Interpret a coefficient in the right units

Assumptions, baselines and forecast errors

What the fitted regression can and cannot identify

Assumptions have different jobs

R², adjusted R² and forecast errors

A baseline makes model error interpretable

Self-check: inference, prediction and timing

Repair the regression sample

OLS does not require Normal predictors

Training and validation measure different things

Inference and Model Selection

Coefficient uncertainty and residual checks

Choose a validation score that reflects the decision

Choose a validation score that reflects the decision · Python

Inference after a model search

Validation is a finite resource

Rebuild the case before model selection

Confidence intervals describe coefficient uncertainty

Read robust coefficient output

An F test asks about several coefficients together

Residual plots test the story behind the fit

Regression residuals can also define a feature

Inspect one cross-section before neutralising

Fit the same OLS form to the feature

Check what the residual transformation guarantees

Specification, redundancy and regularisation

Interactions and nonlinear terms change the question

Best subset is a search with a statistical cost

Multicollinearity inflates coefficient uncertainty

Penalties and scale

Ridge and lasso trade variance for bias

Frozen selection and held-out evidence

Bias, variance and the role of the untouched test

Self-check: selection does not establish causation

VIF diagnoses coefficient redundancy

Nonlinear Models and Loss Classification

Trees, ensembles and feature use

Tree complexity and ensemble reasoning

Rebuild the two targets for nonlinear learning

A regression tree chooses a split by loss reduction

Changing the fitted loss changes the prediction

Changing the fitted loss changes the prediction · Python

Fit a shallow tree and inspect its rule

Classification trees use impurity or log loss

Forests average; boosting adds corrective learners

Compare regression learners on the same validation rows

Importance describes a model’s use of a feature

Event probabilities and classification scores

Probabilities, prevalence and calibration

A 30-day inactivity target is not a 60-day target

A 30-day inactivity target is not a 60-day target · Python

Logistic regression models an event probability

Fit the loss classifier with training-only scaling

Confusion counts depend on a threshold

Ranking and probability quality need separate scores

Decision thresholds and held-out evaluation

Deriving a cost threshold

Interactive threshold and error costs

Probability accuracy and fraud decisions are different

Probability accuracy and fraud decisions are different · Python

Open the held-out test after freezing the rule