The left hand probability and cumulative distributions were made using all 12,000 absolute deviations. This is the distribution that would be used to answer Question 1, and it would appear that an interesting result (at the 5% level for a one-tailed test of significance) is one with an absolute deviation of at least 0.73. This makes the October result (1.0 absolute deviation) appear significant, with only a 0.7% chance of observing a more extreme value in the case of the null hypothesis being true.
But this significance threshold is based on comparing a single month to the distribution of all absolute deviations, whereas we effectively compared 12 when we selected the most extreme value.
To account for multiplicity and to know if the October result is truly significant we need to compare it with the distribution of the most extreme volatility ranks from each of the 1,000 simulated sets. To do this, for each simulation we record just the largest absolute deviation value, and this distribution is shown on the right hand side of Figure 2.
We see that incorporating multiplicity makes the threshold of significance more conservative. To be considered a significant result, an absolute deviation would actually need to be at least 1.1. This makes October’s volatility value no longer significant, with an 8% chance of observing a more extreme value of average volatility rank if the null hypothesis is true.