Distorted Statistics and Performance Tests: Part 3
This part of the series explains Type I and Type II errors as a basis for understanding why the analyst’s conclusion in Part 2 is wrong.
Advanced financial researchers begin by setting up a null hypothesis. The null hypothesis is a statement that the researcher seeks to disprove or fail to disprove. Notice that most good research methods do not seek to include truths, they seek to exclude falsehoods. These researchers should not claim to know the truth, they can only rule out those things that evidence says are false.
For example, an investigator testing to see whether or not an investment strategy works will construct a null hypothesis stating that the strategy does not work. Consider two ancient scholars debating whether the sun or the moon is bigger. They both believe that the moon is very slightly closer to the earth than the sun. The scholar named “Sun” believes that the sun is bigger and the scholar named “Moon” believes that the moon is bigger. Their charts of the paths indicate that a solar eclipse is imminent. Each sets up a null hypothesis that they seek to refute.
Scholar Sun sets up a null hypothesis (that he seeks to disprove) that the moon is bigger. He will refute the null hypothesis if the moon does not fully block out the sun. Scholar Moon sets up a null hypothesis (that she seeks to disprove) that the sun is bigger. She will refute the null hypothesis if the moon fully bocks out the sun.
Here is the amazing result: Scholar Sun fails to refute the hypothesis that the moon is bigger and Scholar Moon refutes the hypothesis that the sun is bigger. The both get it wrong! They both get it wrong because their model was based on a faulty premise – that the distance of the two objects from the earth was similar. But this is an important point: a researcher’s conclusions are only as good as the model on which the test is based.
Scholar Sun’s test results in a Type I error, also known as a false positive, which is when the researcher mistakenly rejects a true hypothesis. Scholar Moon’s test results in a Type II error, also known as a false negative, which is when the researcher mistakenly fails to reject a hypothesis when it is false. This example may seem to be special or even ridiculous due to its obviously erroneous assumption. But it represents the core problem in investment studies. That problem is discussed in Part IV.