Distorted Statistics and Performance Tests: Part 1
Investment analysis, particularly alternative investment analysis, relies on empirical analysis of past returns to form predictions of future risk and return. There are two primary problems with many performance studies: data dredging (also called data snooping), and flawed interpretation of confidence levels.
This first part of a five part series discusses data dredging: when a researcher performs frequent tests on data after having observed some or all of the data. A classic example of data dredging in investments is the testing of multiple investment strategies on a single data set. The investor develops several variables that might predict investment returns and often allows those variables to enter the prediction model with parameters that are determined by the tests themselves. The end result is an over-fitted model that does a great job of explaining the past but has no demonstrated ability to explain the future.
For example, a bond trading system might use variables such as the current level of interest rates, the slope of the yield curve, the flow of new money into bond funds, the federal debt, bank reserves, and several other variables to predict the next move in bond prices. By allowing each of the variables to have one or two parameters that turn the variable on and off under various conditions, it is guaranteed that the trading signals can be retro-fitted to the data such that the trading system appears to mint money.
The investor then uses the trading system to make actual trades and realizes that the system is not working on the new data. So the typical investor then analyzes the successes and failures of the actual trades. By focusing on the trades that had the largest losses and largest profits the investor modifies the signals and begins trading again. Eventually the modified trading system model fails and the investor repeats the process of revising the signals.
The amazing thing is that throughout this process the investor insists that the system is working well. When pushed to show the full track record the investor will say: “Well, I have lost money overall but that was with the old system. The revised system would have worked fine”. Of course! When systems are continuously retrofitted they have a bright past and bad future. It is easy to retrofit a system: any system that is retrofitted to get out of the market in 2000 and 2007 (and get back in after a year or two) will have great past performance.
There are countless examples of retrofitted trading systems that fail when used. The next part details the other bog problem: a flawed understanding of inferential statistics.