When a researcher performs a statistical test on investment returns, the test usually includes a model that specifies how asset returns are assumed to behave in a well-functioning market. For example, the researcher might assume that all strategies run by unskilled manages with similar risk will generate the same average return. Of course the researcher hopes to show (1) that the strategy being tested will earn superior returns, (2) that the hypothesis of equal returns will be rejected, and (3) that the alternative hypothesis is true: the manager being studied is able to generate consistently superior returns.
But the researcher’s conclusions are only as valid as the assumptions on which the test was designed – in particular the assumptions that all of the managers could be properly analyzed by their average performance and that abnormal return performance is normally distributed – a patently false assumption.
The point of this discussion is not to suggest that the scientific methods used are in error. The point is to realize that the results of studies based on inferential statistics – the commonly used method – are only as good as the models assumed, and that claims that something is “99% likely to be true” are 100% false.
Part 5 of this series lays out a detailed numerical example that carefully demonstrates the flawed interpretation of a statistical test.