Distorted Statistics and Performance Tests: Part 5
The example of the sun and moon in Part 3 was somewhat vague and quite unrelated to investment analysis, so this part illustrates the concepts using a more realistic example. Let’s assume all investment traders have equal skill except that 1 out of every 10,000 traders cheats by using inside information. Therefore, the probability of picking a trader at random who uses inside information is 1 in 10,000, or 0.0001. We seek to identify insider traders using a statistical model. We begin with a proper null hypothesis (that we week to disprove) that a given trader is honest (i.e., the trader does not use inside information). An empirical test has been developed that when applied to an honest trader’s transaction record gives a correct answer 99% of the time (that the person does not trade illegally) and a false accusation 1% of the time. In the terms introduced previously, this test has a Type I error rate (i.e., the probability of falsely rejecting the null hypothesis by alleging that an honest trader is cheating) of 1%.
To simplify the problem, let’s assume that when the test is given to a dishonest trader the test always correctly identifies the trader as a cheater (in other words, there is no possibility of a Type II error in which a false null hypothesis is not rejected).
What is the probability that a trader whose transaction record is tested gets a test result indicating that the trader has cheated when in fact the trader has not cheated? Since a 99% confidence interval was used most analysts would conclude that there is a 99% probability that the trader is a cheat. But….
The answer is not 99% — it is only 1%! In this example, only 0.01% of traders actually cheat. Since we are willing to accept a 1% Type I error, this means that even if the test is perfect, 1% of the traders would be falsely accused. That is, from a population of 100,000 traders that are tested, our test would indicate that 1,000 of the traders have cheated. However, since, on average, only 10 traders have actually cheated, this means that 99% of the accused traders are innocent.
In summary, many analysts interpret a significance test as indicating the probability that a test has reached a correct conclusion. So, for example, an analyst using a 95% confidence level might interpret the finding of a non-zero mean or a non-zero coefficient as being 95% indicative that the mean is not zero or the coefficient is not zero. But this would be an erroneous interpretation of the test results.