The mathematics of Drift

Returns — data ingestion and contracts

Simple and log returns

For price P_{i,t}: simple return r_{i,t} = P_{i,t}/P_{i,t-1} − 1; log return ℓ_{i,t} = log(1 + r_{i,t}). Log returns are used wherever stationarity and additivity are required (regime features, wavelets). Simple returns measure outcomes.

The forward-return contract (Axiom 2)

The forward return R_{i,t,h} = P_{i,t+h}/P_{i,t} − 1 peeks into the future. By Axiom 2 it may only appear in evaluation code (IC analysis, backtests, matured view errors) — never in signal generation. The default horizon h = 21 trading days (≈ one month). A CI property test enforces this boundary.

The precomputed FeatureStore

Data is fetched via a fully asynchronous engine from Zerodha Kite, paced exactly at the broker rate limit. All screener math is precomputed at dawn and serialized into memory-mapped Arrow tables. A user request does zero network calls and zero pandas math — the OS maps the file into RAM directly. Fallback: if a snapshot is stale or the universe is custom, the live async pipeline runs without blocking other requests.

Factor Direction & Rank Transform — cross-sectional signal construction

The rank transform — derivation from first principles

We want a function mapping ranks r ∈ {1..N} linearly to scores s ∈ {−1,+1}. Write the linear form s = a·r + b. Two boundary conditions pin the two unknowns:

Subtract the first equation from the second — b cancels:

Substitute back to find b, then combine over a common denominator:

The seven factors

Fama-French 5 factors: Market beta (systematic risk), Size (−log mktcap), Value (B/M ratio), Profitability (ROE), Investment (−asset_growth). Plus Momentum (12-1 month return, skip most recent month per Jegadeesh & Titman 1993) and Quality (composite z-score of ROE, low leverage, low earnings volatility per Asness et al. 2019).

Regime-Aware Factor Weighting — HMM + co-movement graph

The computational problem

For K=3 states and T=252 trading days, brute-force enumeration of all state sequences requires K^T = 3^252 ≈ 10^120 operations. The forward algorithm reduces this to O(K²T) via dynamic programming.

The forward variable (forward-filtered, not smoothed)

Define α_t(k) = P(o₁,...,o_t, s_t = k | λ) — the probability of seeing observations up to t AND being in state k at t.

The co-movement graph upgrade (6-dim embedding)

The most reliable crisis precursor is not the index falling — it is correlations tightening before it falls. Drift 2.0 builds a daily co-movement map of ~11 sectors and injects a 6-dimensional embedding g_t into the HMM emission space: (1) mean pairwise correlation ρ̄_t; (2) Absorption Ratio AR_t = Σ_{k=1}^m θ_k / Σ θ_k (Kritzman et al. 2011); (3) Fiedler value λ₂ of the normalized graph Laplacian — how hard the market network is to cut in two; (4) normalized MST length L̄_t (Mantegna 1999); (5) participation ratio PR_t = (Σ v_{1,i}⁴)⁻¹; (6) lead-lag asymmetry ‖Λ−Λᵀ‖_F / ‖Λ+Λᵀ‖_F.

Regime-blended factor weights

Why StandardScaler is mandatory

log_return, realised_vol, vol_of_vol, and the 6 co-movement features differ in scale by orders of magnitude. Without scaling, Baum-Welch EM collapses all observations into one state. That state's covariance is well-estimated; the minority states have near-zero observations, their covariance matrices become rank-deficient, and the Cholesky decomposition inside the Gaussian emission fails.

Wavelet Scattering Stability — path-stability score

Why the modulus is essential

A complex wavelet ψ_j is a bandpass filter. The raw convolution x★ψ_j oscillates at the centre frequency with zero mean — averaging gives nothing. The modulus |x★ψ_j| strips the oscillation and gives the instantaneous amplitude envelope — slowly varying, non-negative, convolvable at the next scale.

Without modulus: S₂ = x★ψ_j₁★ψ_j₂ is just another bandpass filter (composition of linear operators = linear operator). With modulus: S₂ captures how energy at scale j₁ modulates energy at scale j₂ — a genuinely non-linear feature.

Cauchy vs Morlet

Financial returns are fat-tailed. A Morlet wavelet damps a 10σ return the same as a 3σ return at the same frequency. A Cauchy wavelet retains sensitivity to the 10σ event — exactly the information that matters for systematic risk.

The scattering cascade

Path-stability score (production output)

In the production pipeline, the scattering coefficients are distilled into a single path-stability score in [−1,+1]: +1 = smooth, trending path; −1 = erratic, jumpy behavior. The computation now runs as a single 3D tensor batch operation with a cached filter bank and reflection padding — correcting two bugs present in the legacy per-ticker loop implementation.

Factor Reliability — IC, ICIR, and signal combination

Why Spearman, not Pearson

A single earnings shock (+30% on one stock, ±2% everywhere else) produces a near-zero Pearson IC even if the factor correctly ranked all other stocks. Spearman rank correlation is Pearson correlation of the rank vectors — a single outlier gets rank N, not a 15σ influence on the correlation.

ICIR as signal Sharpe ratio

IC-weighted combination

Black-Litterman — self-calibrating Bayesian posterior

Step 1 — Reverse optimisation

CAPM says market-cap weights w_mkt are mean-variance optimal. Running the formula backwards gives the implied equilibrium expected returns:

Step 2 — Views as active tilts above equilibrium (corrected)

The legacy implementation set q = Z · scale(μ_eq), which dragged expected-return levels toward zero. The corrected formula treats the composite alpha Z as an active tilt above the equilibrium prior:

Step 3 — Endogenous view uncertainty Ω (the key upgrade)

In the legacy model, Ω = τ · diag(PΣPᵀ) was a single hand-set constant (He-Litterman uncertainty). Drift 2.0 makes Ω endogenous — computed from the model's own measured IC accuracy and realized view errors:

Step 4 — Posterior (conjugate Gaussian update)

Hierarchical Risk Parity — clustering-based base allocation

Step 1 — Distance matrix

Step 2 — Ward linkage and quasi-diagonalisation

Ward linkage minimises within-cluster variance at each merge. The dendrogram leaf order (quasi-diagonalisation) reorders the covariance matrix so similar assets are adjacent.

Step 3 — Recursive bisection

CVaR — the Rockafellar-Uryasev linear programme

The non-smooth problem

The max() makes this non-smooth. Standard gradient methods fail. Rockafellar-Uryasev (2000) showed it reformulates as a linear programme:

Covariance & Risk Model — Ledoit-Wolf shrinkage and BARRA decomposition

Why sample covariance fails

An N×N covariance matrix has N(N+1)/2 unique parameters. When T approaches N, the condition number (max/min eigenvalue) explodes. Inverting such a matrix amplifies noise by the condition number — 10³ or more for a 100-asset portfolio with 2 years of daily data.

Ledoit-Wolf shrinkage

BARRA decomposition

Backtest Validation — walk-forward, PSR, and DSR

Sharpe ratio variance under non-normality

Probabilistic Sharpe Ratio

Deflated Sharpe Ratio

When N strategies are tested and the best is selected, E[max SR] under the null rises with N. DSR adjusts the benchmark SR* to account for selection bias: