Distorted Statistics and Performance Tests: Part 1

  • September 13, 2017

    Investment analysis, particularly alternative investment analysis, relies on empirical analysis of past returns to form predictions of future risk and return. There are two primary problems with many performance studies: data dredging (also called data snooping), and flawed interpretation of confidence levels.

    This first part of a five part series discusses data dredging: when a researcher performs frequent tests on data after having observed some or all of the data. A classic example of data dredging in investments is the testing of multiple investment strategies on a single data set. The investor develops several variables that might predict investment returns and often allows those variables to enter the prediction model with parameters that are determined by the tests themselves. The end result is an over-fitted model that does a great job of explaining the past but has no demonstrated ability to explain the future.

    For example, a bond trading system might use variables such as the current level of interest rates, the slope of the yield curve, the flow of new money into bond funds, the federal debt, bank reserves, and several other variables to predict the next move in bond prices. By allowing each of the variables to have one or two parameters that turn the variable on and off under various conditions, it is guaranteed that the trading signals can be retro-fitted to the data such that the trading system appears to mint money.

    The investor then uses the trading system to make actual trades and realizes that the system is not working on the new data. So the typical investor then analyzes the successes and failures of the actual trades. By focusing on the trades that had the largest losses and largest profits the investor modifies the signals and begins trading again. Eventually the modified trading system model fails and the investor repeats the process of revising the signals.

    The amazing thing is that throughout this process the investor insists that the system is working well. When pushed to show the full track record the investor will say: “Well, I have lost money overall but that was with the old system. The revised system would have worked fine”. Of course! When systems are continuously retrofitted they have a bright past and bad future. It is easy to retrofit a system: any system that is retrofitted to get out of the market in 2000 and 2007 (and get back in after a year or two) will have great past performance.

    There are countless examples of retrofitted trading systems that fail when used. The next part details the other bog problem: a flawed understanding of inferential statistics.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>



1256 contracts, 2010, 401K, adp report, Advisors, Alexandra Twin, alternative investments, annuity, asset allocation, asset classes, asset management, Associated Press, away, banks, Barrons Top 100, BCA, BCA. Biltmore Capital, Ben Bernanke, Best Week, bias, Biltmore, Biltmore Capital, Biltmore Capital Advisors, bond market, Brand, Brand Recognition, Business, Call, CFP Princeton, changes in market price, Chief Investment Officer, closed end funds, CNBC, CNBC Halftime Report, CNN, CNN Money, CNNMoney, CNNMoney.com, collar stock, college grads, Consumer Confidence Report, correlations, Covered, Covered Call Options, credit ratings, customized investment strategies, Dean, donald chambers, Dow Jones, dr don chambers, drop, Early, easy trading, economic policy, economy, Edge, efficiency, election, end, equilibrium, equity trading, etf, etf trading, ETFs, euro, Europe, Europe debt, European debt crisis, family office approach, federal income tax, federal investment income tax, Financial, financial advisor, financial crises, financial distress, financial management, financial security, financial strategies, Ford, Fox, Fox Business, fund manager, futures contracts, Gains, General Motors, Global, global market, Global stocks, groupon, halftime report, Herbert Lash, high income tax, home country, Home Sales, housing market, IDEAS, income tax, income taxes, initial public offering, Instant View, investment bias, investment income, investment returns, investment risk, investment strategies, Investors, IPOs, IRA, Jilian Mincer, Jonathan Cheng, KANA INAGAKI, key, large losses, long risk exposure, Los Angeles Times, make money, making investment money, making money last, Manufacture, Manufacturing, Market, market news, market price, market volatility, Markets, marketwatch, MICHELE MAATOUK, Mike Miliard, Molly Vernon, money, Money Manager, money managers, money strategy, municipal bond interest, municipal bonds, nassau club, New Jersey Advisors, New York Times, NJ, NJ advisors, NJ financial advisors, NJ money manager, NJ wealth advisor, NJ Wealth Advisors, obamacare, oil slide, Options, options strategies, Outlook, outperformance, Pending, personal financial services, personal risk analysis, Play, portfolio manager, Potfolio Manager, Princeton, Princeton Advisors, Princeton asset management, princeton financial advisors, Princeton Money Managers, Princeton wealth advisor, private wealth management, rally, recession, Recognition, registered investment advisor, Registered Investment Advisory Firm, retirement, retirement strategy, return, returns, Reuters, reward, RIA, RIA Princeton, Risk, risk exposure, riskier, rmd, Roth IRA, safer, saving taxes, savings, SEC-registered, Seeking Alpha, short risk exposure, Shudder, signals, skype, social security, social security benefits rules, star ledger, state income tax, Stephen Bernard, Stock, stock market, stock market returns, stock market winners, stock option strategies, stock price, stock prices, stock research, stock strategy, Stock Volatility, Stocks, structured notes, Stuart Day School, swine flu, tail risk, Tax Advantaged Investments, tax free investments, tax savings, tax strategy, taxes, taxes for social security, taxes on investment income, The Dean, The Wall Street Journal, Thomson, Thomson Reuters, Tick, Tim Ralph, Timothy Ralph, Today, Tony Roth, Trading, tricks, Tyler, tyler vernon, U.S., U.S. dollar gains, U.S. Stock, unemployment, USA, USA Today, VIX, volatility, Wall Street, wealth management, wealth manager, wells fargo, world market, worth, Yahoo, Yahoo Financial,