Screening Using False-Discovery Rates

1. Motivating Example

Jegadeesh and Titman (1993) show that, if you rank stocks according to their returns over the previous $12$ months, then the past winners will outperform the past losers by $1.5{\scriptstyle \%}$ per month over the next $3$ months. But, the authors don’t just test this particular strategy. They also test strategies that rank stocks over the previous $3$ , $6$ , and $9$ months and strategies which hold stocks for the next $6$ , $9$ , and $12$ months, too. Clearly, if they test enough hypotheses, then some of these tests are going to appear statistically significant by pure chance. To address this concern in the original paper, the authors use the Bonferroni method.

This post shows how to use an alternative method—namely, controlling the false-discovery rate—to identify statistically significant results when testing multiple hypotheses.

2. Bonferroni Method

First, here’s the logic behind the Bonferroni method. Suppose you want to run $N \gg 1$ different hypothesis tests, $\{ \, h_1, \, h_2, \, \ldots, \, h_N \, \}$ . Let $h_n = 0$ if the $n$ th null hypothesis is true and $h_n = 1$ if the $n$ th null hypothesis is false (i.e., should be rejected). Let $p_n$ denote the p-value associated with some test statistic for the $n$ th hypothesis. If there were just one test, then we should simply reject the null whenever $p_1 \leq 0.05$ . But, if there are many tests, then this no longer works. If you look at a lot of hypotheses, then $5{\scriptstyle \%}$ of the p-values should be less than $0.05$ even if $h_n = 0$ for all of them. The Bonferroni method suggests correcting this problem by lowering the p-value associated with the $5{\scriptstyle \%}$ significance level and only rejecting the null hypothesis when

(1) $\begin{align*} p_n \leq {\textstyle \frac{1}{N}} \cdot 0.05. \end{align*}$

i.e., if there are $N = 10$ hypothesis tests, then only reject the null at the $5{\scriptstyle \%}$ significance level when the p-value is less than $0.005$ rather than $0.05$ .

This is a nice start, but it turns out that the Bonferroni method is way too strict. Imagine drawing samples of $10$ observations from $N$ different normal distributions. All samples have the same standard deviation, $\sigma = 1$ , but not all of the sample have the same mean. $20{\scriptstyle \%}$ have a mean of $\mu = 1$ , and the rest have a mean of $\mu = 0$ . The figure below shows that, if we use the Bonferroni method to identify which of the $N = 100$ samples have a non-zero mean, then we’re only going to choose $2$ samples. But, by construction, we know that $20$ samples had a non-zero mean! We should be rejecting the null $10$ -times more often!

3. False-Discovery Rate

Now, let’s talk about false-discovery rates. Define $R(0.05) = \sum_{n=1}^N 1_{\{p_n < 0.05\}}$ as the total number hypotheses that you reject at the $5{\scriptstyle \%}$ significance level. Similarly, define $R_f(0.05) = \sum_{n=1}^N 1_{\{p_n < 0.05\}} \times 1_{\{ h_n = 0 \}}$ as the number of hypotheses that you reject at the $5{\scriptstyle \%}$ significance level where the null was actually true—i.e., these are false rejections. The false-discovery rate is then

(2) $\begin{align*} \mathit{FDR} = \mathrm{E}\left[ \, \sfrac{R_f}{R} \, \right]. \end{align*}$

Let’s return to the numerical example above to clarify this definition. Suppose we had a test that identified all $20$ cases where the sample mean was $\mu = 1$ , $R = 20$ . If we wanted a false-discovery rate $\mathit{FDR} \leq 0.10$ , then this test could produce at most $2$ false rejections, $R_f \leq 2$ . If the test identified only half of the $20$ cases where the sample mean was $\mu =1$ , $R = 10$ , then this test could produce at most $1$ one false rejection, $R_f \leq 1$ .

Benjamini and Hochberg (1995) first introduced the idea that you could use the false-discovery rate to adjust statistical-significance tests when exploring multiple hypotheses. Here’s their recipe. First, run all of your tests and order the resulting p-values,

(3) $\begin{align*} p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(n)} \leq \cdots \leq p_{(N)}. \end{align*}$

Then, for a given false-discovery rate, $x$ , define $n_x$ as

(4) $\begin{align*} n_x = \max_{n \in N} \, \left\{ \, n \, : \, p_{(n)} \leq {\textstyle \frac{n}{N} \cdot x} \, \right\}. \end{align*}$

Benjamini and Hochberg (1995) showed that, if you reject any null hypothesis where

(5) $\begin{align*} p_n \leq p_{(n_x)}, \end{align*}$

then $\mathit{FDR} \leq x$ , guaranteed. If we apply the false-discovery-rate procedure to the same numerical example from above using the $x = 0.05$ threshold, then we see that as the number of hypotheses gets large, $N \to \infty$ , the fraction of rejected null hypotheses hovers around $0.06$ . Improvement! It’s no longer shrinking to $0$ . Notice that neither method allows us to pick out the full $20{\scriptstyle \%}$ of null hypotheses that should be rejected in the simulation.

4. Why’s This So?

It’s pretty clear how the Bonferroni method works. If there are lots of hypotheses and you are worried about rejecting the null by pure chance, then just make it harder to reject the null. i.e., lower the threshold for significant p-values. It’s much less clear, though, how the false-discovery-rate screening process works. All you get is a recipe and a guarantee. If you do the list of things prescribed by Benjamini and Hochberg (1995), then you’ll falsely reject no more than $x{\scriptstyle \%}$ of your null hypotheses. Let’s explore the proof of this result from Storey, Taylor and Siegmund (2003) to better understand where this guarantee comes from.

What does it mean to have a useful test? Well, if $h_n = 0$ (null is true), then $p_n$ is drawn randomly from a uniform distribution, $\mathrm{U}(0,\,1)$ . However, if you have a useful test statistic, then if $h_n = 1$ (null is false), then $p_n$ is drawn from some other distribution, $\mathrm{A}(0,\,1)$ that is more concentrated around $0$ . The distribution of p-values is then given by

(6) $\begin{align*} \mathrm{G} = \pi \cdot \mathrm{U} + (1 - \pi) \cdot \mathrm{A}. \end{align*}$

If we reject all p-value less than $x$ , then with a little bit of algebra we can see that

(7) $\begin{align*} \mathit{FDR}(x) = \mathrm{E}\left[ \, \frac{ {\textstyle \sum_{n=1}^N} 1_{\{ p_n \leq x\}} \times 1_{\{ h_n = 0 \}} }{ {\textstyle \sum_{n=1}^N} 1_{\{ p_n \leq x \}} } \, \right] &= \frac{ \mathrm{E}\left[ \, {\textstyle \sum_{n=1}^N} 1_{\{ p_n \leq x\}} \times 1_{\{ h_n = 0 \}} \, \right] }{ \mathrm{E}\left[ \, {\textstyle \sum_{n=1}^N} 1_{\{ p_n \leq x \}} \, \right] } + \mathrm{O}\left( {\textstyle \frac{1}{\sqrt{N}}} \right) \\ &= \frac{ N \cdot \mathrm{Pr}( p_n \leq x | h_n = 0 ) \cdot \mathrm{Pr}(h_n = 0) }{ N \cdot \mathrm{G}(x) } + \mathrm{O}\left( {\textstyle \frac{1}{\sqrt{N}}} \right) \end{align*}$

where the last step uses the fact that $R(x) = {\textstyle \sum_{n=1}^N} 1_{\{ p_n \leq x \}} = N \cdot \mathrm{G}(x)$ and $\mathrm{O}( \sfrac{1}{\sqrt{N}})$ denotes “big-O” notation. Since p-values are drawn from a uniform distribution when the null hypothesis is true, we know that $x = \mathrm{Pr}( p_n \leq x | h_n = 0 )$ . Thus, we can simplify even further:

(8) $\begin{align*} \mathit{FDR}(x) &= \frac{ x \cdot \mathrm{Pr}(h_n = 0) }{ \mathrm{G}(x) } + \mathrm{O}\left( {\textstyle \frac{1}{\sqrt{N}}} \right). \end{align*}$

Now comes the trick. $x$ can be anything we want between $0$ and $1$ . So, let’s choose $x$ as one of the ordered p-values, $p_{(n)}$ . If we do this, then $\mathrm{G}(p_{(n)}) = \sfrac{n}{N}$ and

(9) $\begin{align*} \mathit{FDR}(p_{(n)}) \approx \frac{p_{(n)} \cdot \mathrm{Pr}(h_n = 0) \cdot N}{n} \leq \frac{p_{(n)} \cdot N}{n}. \end{align*}$

If we set the right-hand side equal to the false-discovery-rate tolerance, $0.05$ , and solve for $p_{(n)}$ , then we get the threshold value for $p_{(n)}$ in Benjamini and Hochberg (1995),

(10) $\begin{align*} p_{(n)} = {\textstyle \frac{n}{N}} \times 0.05. \end{align*}$

If we only reject hypotheses where $p_n \leq {\textstyle \frac{n}{N}} \times 0.05$ , then our false-discovery rate is capped at $5{\scriptstyle \%}$ .