Five hundred assets, T days

What happens to portfolio construction when the universe is as wide as the history is long.

Portfolio methods are usually demonstrated on a dozen assets and decades of data. Real universes are wide: hundreds or thousands of names, and often less history than that — new listings, regime changes, or simply a survivorship-clean window. A covariance matrix on n assets has n(n+1)/2 free parameters; with n wider than the number of observations T, the sample covariance is singular, and every allocator built on Σ⁻¹ stops being defined — not approximately, exactly.

This page watches that happen. The market is synthetic on purpose: 500 names in 10 sectors, each driven by one market factor and one sector factor, so the true covariance — and therefore the true minimum-variance portfolio and its variance — is known exactly. Every "× true GMV" below has an exact denominator, which no real dataset can give you. One simulated history of 2000 days; the slider shortens the estimation window.

Try it: shrink the window

What to look at

The sample minimum-variance portfolio w ∝ S⁻¹1 passes through three regimes. With plenty of data (n/T = 0.25) it sits at 1.4× the true minimum variance — when the data supports the inversion, invert. As T falls toward n the out-of-sample variance climbs — 2× at n/T = 0.5, 16× at 0.95, 91× at 0.99 — while the leverage the portfolio takes rises from 4× to 40×. Past n/T = 1 the matrix is singular and what the solver returns is decided by rounding error: the plotted values jump across orders of magnitude from one window to the next, because they are noise, not estimates.

Shrinkage is the standard repair, so it is on the chart too. Ledoit–Wolf minimum variance is defined at every n/T and does well here — 1.3× to 2.0× across the whole sweep, better variance than anything long-only on this page. What it does not repair is the character of the position: it holds roughly 4.5× gross leverage with substantial short legs at every window length, and its weights remain estimates of an inverse. Where a levered long–short mandate is available and variance is the whole objective, shrunk minimum variance is the right tool, and this page says so.

The race line does not move. Weights are the winning probabilities of a Thurstone race run with a rank-12 factor decomposition of the sample correlation — the factors=k low-rank transport, O(Mnk) per update, with no matrix inverted anywhere — so nothing changes qualitatively when T crosses n: long-only, leverage 1, out-of-sample variance flat at about 3.3× the true minimum. Within the long-only, unlevered class its comparison is equal weight at 3.5×.

The honest comparison to keep in mind: equal weight sits at 3.5× on this market, so the race buys only a few percent of variance over the naive portfolio here. Its claim in high dimension is not that it beats a well-estimated GMV — nothing on this page does — but that it remains defined, stable, and correlation-aware in the regime where inversion-based weights are noise. The same low-rank transport is what makes the estimator usable at Russell-3000 scale, where n/T > 1 is not a stress test but the everyday condition.

Reproducible: experiments/gen_highdim_data.py builds the market, the minimum-variance sweep, and the factor decompositions with the allocation package; the in-browser race is the forward pass of transport_weights_lowrank (see ThurstonePortfolio(factors=k) in allocation/thurstone.py) on a fixed ensemble of 8,192 scenarios.