Trolling for Investment Strategies
If one searches long enough in a large enough pool of data, statistically significant correlations will inevitably be found. That doesn’t mean these relationships are meaningful or the result of anything more than pure chance. A recent paper by a group of mathematicians addresses the practice of investment managers trolling historical data to identify profitable strategies. This is a case where the tired disclaimer that “past performance is no guarantee of future results” should be heeded. Investing must be done through the windshield rather than through the rear-view mirror.
The paper’s abstract:
Recent computational advances allow investment managers to methodically search through thousands or even millions of potential options for a proﬁtable investment strategy. In many instances, the resulting strategy involves a pseudo-mathematical argument, which is spuriously validated through a simulation of its historical performance (also called a ‘backtest’). We prove that high performance is easily achievable after backtesting a relatively small number of alternative strategy conﬁgurations, a practice we denote “backtest overﬁtting.” The higher the number of conﬁgurations tried, the greater is the probability that the backtest is overﬁt. Because ﬁnancial analysts rarely report the number of conﬁgurations tried for a given backtest, investors cannot evaluate the degree of overﬁtting in most investment claims and analysis. The implication is that investors can be easily misled into allocating capital to strategies that appear to be mathematically sound and empirically supported by a backtest. This practice is particularly pernicious, because due to the nature of ﬁnancial time series, backtest overﬁtting has a detrimental eﬀect on the strategy’s future performance.
We particularly enjoyed this call to arms within the paper:
Historically scientists have led the way in exposing those who utilize pseudoscience to extract a commercial beneﬁt. As early as the 18th century, physicists exposed the nonsense of astrologers. Yet mathematicians in the 21st century have remained disappointingly silent with the regards to those in the investment community who, knowingly or not, misuse mathematical techniques such as probability theory, statistics and stochastic calculus. Our silence is consent, making us accomplices in these abuses.