Distorted Statistics and Performance Tests: Part 2

  • September 20, 2017

    The second major problem with statistical tests as performed in finance and economics is a fundamental misunderstanding of inferential statistics. Researchers perform studies based on models. They attribute great importance to variables that are statistically significant. But most researchers completely misstate what the results mean.

    A typical statistical analysis in investment analysis uses regression techniques to explore whether one variable is caused by another variable such as whether Federal Reserve announcements affect market returns in the days after they are officially made public. The regression is usually performed at some level of confidence such as 99% (i.e., a significance level of 1%). If a variable is found to be statistically significant it is deemed as being very likely to be important. If a 99% confidence level is used, analysts usually claim that it is 99% likely that the variable is important. That is not true.

    For example, assume that a hedged fund analyst performs a test with a 99% level of confidence as to whether or not stock returns in the days after a Fed announcement are higher than in other days of the year. Suppose that the test generates a statistically significant result indicating that the returns are higher in the days immediately after Fed meetings at a confidence level of 99%. The incorrect conclusion that is often formed regarding such a result is that there is a 99% probability that Fed announcements are followed by higher returns. In fact, the hedge fund analyst has no reasonable basis for this conclusion – even though most researchers do so anyway. This interpretation is a false belief based on erroneous statistical interpretations.

    To explain this important concept carefully the next part in this series describes two types of errors: Type I errors and Type II errors. It takes some work to understand these two types of errors – but it leads to the important benefit of understanding what these types of statistical tests do and do not mean.

    So if being statistically significant at the 99% level does not mean that there is a 99% chance that the relationship is true, what does it mean? It means that if we assume that the model is correctly specified (a big if that we usually have little reason to believe), there is only a 1% chance that the results could have occurred due to chance. The next Part in this series explores this concept in greater detail.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>



1256 contracts, 2010, 401K, adp report, Advisors, Alexandra Twin, alternative investments, annuity, asset allocation, asset classes, asset management, Associated Press, away, banks, Barrons Top 100, BCA, BCA. Biltmore Capital, Ben Bernanke, Best Week, bias, Biltmore, Biltmore Capital, Biltmore Capital Advisors, bond market, Brand, Brand Recognition, Business, Call, CFP Princeton, changes in market price, Chief Investment Officer, closed end funds, CNBC, CNBC Halftime Report, CNN, CNN Money, CNNMoney, CNNMoney.com, collar stock, college grads, Consumer Confidence Report, correlations, Covered, Covered Call Options, credit ratings, customized investment strategies, Dean, donald chambers, Dow Jones, dr don chambers, drop, Early, easy trading, economic policy, economy, Edge, efficiency, election, end, equilibrium, equity trading, etf, etf trading, ETFs, euro, Europe, Europe debt, European debt crisis, family office approach, federal income tax, federal investment income tax, Financial, financial advisor, financial crises, financial distress, financial management, financial security, financial strategies, Ford, Fox, Fox Business, fund manager, futures contracts, Gains, General Motors, Global, global market, Global stocks, groupon, halftime report, Herbert Lash, high income tax, home country, Home Sales, housing market, IDEAS, income tax, income taxes, initial public offering, Instant View, investment bias, investment income, investment returns, investment risk, investment strategies, Investors, IPOs, IRA, Jilian Mincer, Jonathan Cheng, KANA INAGAKI, key, large losses, long risk exposure, Los Angeles Times, make money, making investment money, making money last, Manufacture, Manufacturing, Market, market news, market price, market volatility, Markets, marketwatch, MICHELE MAATOUK, Mike Miliard, Molly Vernon, money, Money Manager, money managers, money strategy, municipal bond interest, municipal bonds, nassau club, New Jersey Advisors, New York Times, NJ, NJ advisors, NJ financial advisors, NJ money manager, NJ wealth advisor, NJ Wealth Advisors, obamacare, oil slide, Options, options strategies, Outlook, outperformance, Pending, personal financial services, personal risk analysis, Play, portfolio manager, Potfolio Manager, Princeton, Princeton Advisors, Princeton asset management, princeton financial advisors, Princeton Money Managers, Princeton wealth advisor, private wealth management, rally, recession, Recognition, registered investment advisor, Registered Investment Advisory Firm, retirement, retirement strategy, return, returns, Reuters, reward, RIA, RIA Princeton, Risk, risk exposure, riskier, rmd, Roth IRA, safer, saving taxes, savings, SEC-registered, Seeking Alpha, short risk exposure, Shudder, signals, skype, social security, social security benefits rules, star ledger, state income tax, Stephen Bernard, Stock, stock market, stock market returns, stock market winners, stock option strategies, stock price, stock prices, stock research, stock strategy, Stock Volatility, Stocks, structured notes, Stuart Day School, swine flu, tail risk, Tax Advantaged Investments, tax free investments, tax savings, tax strategy, taxes, taxes for social security, taxes on investment income, The Dean, The Wall Street Journal, Thomson, Thomson Reuters, Tick, Tim Ralph, Timothy Ralph, Today, Tony Roth, Trading, tricks, Tyler, tyler vernon, U.S., U.S. dollar gains, U.S. Stock, unemployment, USA, USA Today, VIX, volatility, Wall Street, wealth management, wealth manager, wells fargo, world market, worth, Yahoo, Yahoo Financial,