Distorted Statistics and Performance Tests: Part 5

  • October 11, 2017

    The example of the sun and moon in Part 3 was somewhat vague and quite unrelated to investment analysis, so this part illustrates the concepts using a more realistic example. Let’s assume all investment traders have equal skill except that 1 out of every 10,000 traders cheats by using inside information. Therefore, the probability of picking a trader at random who uses inside information is 1 in 10,000, or 0.0001. We seek to identify insider traders using a statistical model. We begin with a proper null hypothesis (that we week to disprove) that a given trader is honest (i.e., the trader does not use inside information). An empirical test has been developed that when applied to an honest trader’s transaction record gives a correct answer 99% of the time (that the person does not trade illegally) and a false accusation 1% of the time. In the terms introduced previously, this test has a Type I error rate (i.e., the probability of falsely rejecting the null hypothesis by alleging that an honest trader is cheating) of 1%.

    To simplify the problem, let’s assume that when the test is given to a dishonest trader the test always correctly identifies the trader as a cheater (in other words, there is no possibility of a Type II error in which a false null hypothesis is not rejected).

    What is the probability that a trader whose transaction record is tested gets a test result indicating that the trader has cheated when in fact the trader has not cheated? Since a 99% confidence interval was used most analysts would conclude that there is a 99% probability that the trader is a cheat. But….

    The answer is not 99% — it is only 1%! In this example, only 0.01% of traders actually cheat. Since we are willing to accept a 1% Type I error, this means that even if the test is perfect, 1% of the traders would be falsely accused. That is, from a population of 100,000 traders that are tested, our test would indicate that 1,000 of the traders have cheated. However, since, on average, only 10 traders have actually cheated, this means that 99% of the accused traders are innocent.

    In summary, many analysts interpret a significance test as indicating the probability that a test has reached a correct conclusion. So, for example, an analyst using a 95% confidence level might interpret the finding of a non-zero mean or a non-zero coefficient as being 95% indicative that the mean is not zero or the coefficient is not zero. But this would be an erroneous interpretation of the test results.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>



1256 contracts, 2010, 401K, adp report, Advisors, Alexandra Twin, alternative investments, annuity, asset allocation, asset classes, asset management, Associated Press, away, banks, Barrons Top 100, BCA, BCA. Biltmore Capital, Ben Bernanke, Best Week, bias, Biltmore, Biltmore Capital, Biltmore Capital Advisors, bond market, Brand, Brand Recognition, Business, Call, CFP Princeton, changes in market price, Chief Investment Officer, closed end funds, CNBC, CNBC Halftime Report, CNN, CNN Money, CNNMoney, CNNMoney.com, collar stock, college grads, Consumer Confidence Report, correlations, Covered, Covered Call Options, credit ratings, customized investment strategies, Dean, donald chambers, Dow Jones, dr don chambers, drop, Early, easy trading, economic policy, economy, Edge, efficiency, election, end, equilibrium, equity trading, etf, etf trading, ETFs, euro, Europe, Europe debt, European debt crisis, family office approach, federal income tax, federal investment income tax, Financial, financial advisor, financial crises, financial distress, financial management, financial security, financial strategies, Ford, Fox, Fox Business, fund manager, futures contracts, Gains, General Motors, Global, global market, Global stocks, groupon, halftime report, Herbert Lash, high income tax, home country, Home Sales, housing market, IDEAS, income tax, income taxes, initial public offering, Instant View, investment bias, investment income, investment returns, investment risk, investment strategies, Investors, IPOs, IRA, Jilian Mincer, Jonathan Cheng, KANA INAGAKI, key, large losses, long risk exposure, Los Angeles Times, make money, making investment money, making money last, Manufacture, Manufacturing, Market, market news, market price, market volatility, Markets, marketwatch, MICHELE MAATOUK, Mike Miliard, Molly Vernon, money, Money Manager, money managers, money strategy, municipal bond interest, municipal bonds, nassau club, New Jersey Advisors, New York Times, NJ, NJ advisors, NJ financial advisors, NJ money manager, NJ wealth advisor, NJ Wealth Advisors, obamacare, oil slide, Options, options strategies, Outlook, outperformance, Pending, personal financial services, personal risk analysis, Play, portfolio manager, Potfolio Manager, Princeton, Princeton Advisors, Princeton asset management, princeton financial advisors, Princeton Money Managers, Princeton wealth advisor, private wealth management, rally, recession, Recognition, registered investment advisor, Registered Investment Advisory Firm, retirement, retirement strategy, return, returns, Reuters, reward, RIA, RIA Princeton, Risk, risk exposure, riskier, rmd, Roth IRA, safer, saving taxes, savings, SEC-registered, Seeking Alpha, short risk exposure, Shudder, signals, skype, social security, social security benefits rules, star ledger, state income tax, Stephen Bernard, Stock, stock market, stock market returns, stock market winners, stock option strategies, stock price, stock prices, stock research, stock strategy, Stock Volatility, Stocks, structured notes, Stuart Day School, swine flu, tail risk, Tax Advantaged Investments, tax free investments, tax savings, tax strategy, taxes, taxes for social security, taxes on investment income, The Dean, The Wall Street Journal, Thomson, Thomson Reuters, Tick, Tim Ralph, Timothy Ralph, Today, Tony Roth, Trading, tricks, Tyler, tyler vernon, U.S., U.S. dollar gains, U.S. Stock, unemployment, USA, USA Today, VIX, volatility, Wall Street, wealth management, wealth manager, wells fargo, world market, worth, Yahoo, Yahoo Financial,