20 December 2013
- 12 minute read

The performance distribution, as measured by the Sharpe ratio, of skill-free trading strategies, over 23 years.In this example, the probability of obtaining a Sharpe ratio better than 0.6 is only 0.002. This is the “p-value".

30 data points generated from a first-order polynomial (two parameters), subsequently fitted with polynomials of order 1, 4, and 9. The higher order polynomial will have a lower average residual between the data and the model fit. But it is unlikely to fit new data.

The model parameters were chosen based on the period 1980 to 2004. When applied to the out-of-sample periods the performance is significantly worse than during the in-sample period.

The performance distribution of individual skill-free trading strategies, over 23 years is shown in red. The performance distribution of a skill-free trading strategy cherry-picked as the best performer from a set of 100 is shown in blue. In this example, the probability of obtaining a Sharpe ratio better than 0.6 is only 0.002 in the individual distribution but increases to 0.15 in the “best-of-100" distribution.

[1] Trouble at the lab, The Economist, 23-27, Oct. 19th-25th, 2013.

[2] D. Sarewitz, Beware the creeping cracks of bias, Nature, 485, 149, 2012.

[3] J. P. A. Ioannidis, Why Most Published Research Findings Are False, Public Library of Science: Medicine, 8, 2005.

[4] J. P. Simmons, L. D. Nelson, U. Simonsohn, False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant, Psychological Science, 22, 1359-1366, 2011.

[5] C. M. Bennett, et al., Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction, Human Brain Mapping Conference, San Francisco, Jun. 2009.

[6] J. P. A. Ioannidis, Contradicted and initially stronger effects in highly cited clinical research, Journal of the American Medical Association, 294, 218-228, 2005.

[7] E. J. Masicampo, D. R. Lalande, A peculiar prevalence of p values just below .05, The Quarterly Journal of Experimental Psychology, DOI: 10.1080/17470218.2012.711335, 2012.

[8] C. Doucouliagos, T. D. Stanley, Are all economic facts greatly exaggerated? Theory competition and selectivity, Journal of Economic Surveys, 27, 316-339, 2013.

[9] A. Gerber, N. Malhotra, Do statistical reporting standards affect what is published? Publication bias in two leading political science journals, Quarterly Journal of Political Science, 3, 313-326, 2008.

[10] J. M. Dickson, S. Padmawar, S. Hammer, Joined at the hip: ETF and index development, Vanguard research, 2012.

[11] Winton Research, The Hypothetical Performance of CTAs, 2013.

[12] C. Chambers et al., Trust in science would be improved by study pre-registration, The Guardian, Letters, Jun. 5, 2013.

[13] E. J. Wagenmakers et al., An agenda for purely confirmatory research, Perspectives on Psychological Science, 7, 632-638, 2012.

[14] E. F. Fama, K. R. French, Luck Versus Skill in the Cross Section of Mutual Fund Returns, Journal of Finance, 5, 1915-1947, 2010.