The logic of the Fisher method to combine P-values

Consider a set of k independent tests, each of these to test a certain null hypothesis \mathcal{H}_{0|i}, i=\{1, 2, \ldots, k\}. For each test, a significance level P_{i}, i.e., a p-value, is obtained. All these p-values can be combined into a joint test whether there is a global effect, i.e., if a global null hypothesis \mathcal{H}_0 can be rejected.

There are a number of ways to combine these independent, partial tests. The Fisher method is one of these, and is perhaps the most famous and most widely used. The test was presented in Fisher’s now classical book, Statistical Methods for Research Workers, and was described rather succinctly:

When a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance. It is sometimes desired, taking account only of these probabilities, and not of the detailed composition of the data from which they are derived, which may be of very different kinds, to obtain a single test of the significance of the aggregate, based on the product of the probabilities individually observed.
The circumstance that the sum of a number of values of \chi^{2} is itself distributed in the \chi^{2} distribution with the appropriate number of degrees of freedom, may be made the basis of such a test. For in the particular case when n=2, the natural logarithm of the probability is equal to \frac{1}{2}\chi^{2}. If therefore we take the natural logarithm of a probability, change its sign and double it, we have the equivalent value of \chi^{2} for 2 degrees of freedom. Any number of such values may be added together, to give a composite test, using the Table of \chi^{2} to examine the significance of the result. — Fisher, 1932.

The test is based on the fact that the probability of rejecting the global null hypothesis is related to intersection of the probabilities of each individual test, \prod_{i} P_{i}. However, \prod_{i} P_{i} is not uniformly distributed, even if the null is true for all partial tests, and cannot be used itself as the joint significance level for the global test. To remediate this fact, some interesting properties and relationships among distributions of random variables were exploited by Fisher and embodied in the succinct excerpt above. These properties are discussed below.

The logarithm of uniform is exponential

The cumulative distribution function (cdf) of an exponential distribution is:

F(x)=1- e^{-\lambda x}

where \lambda is the rate parameter, the only parameter of this distribution. The inverse cdf is, therefore, given by:

x = -\dfrac{1}{\lambda}\ln(1-F(x))

F(x)=P is a random variable uniformly distributed in the interval [0, 1], and so is 1-P, and it is immaterial to differ between them. As a consequence, the previous equation can be equivalently written as:

x = -\dfrac{1}{\lambda}\ln(P)

where P \sim \mathcal{U}(0,1), which highlights the fact that the negative of the natural logarithm of a random variable distributed uniformly between 0 and 1 follows an exponential distribution with rate parameter \lambda=1.

An exponential with rate 1/2 is chi-squared

The cdf of a chi-squared distribution with \nu degrees of freedom, i.e. \chi^{2}_{\nu}, is given by:

F(x; \nu) = \dfrac{\int_{0}^{x/2} t^{\frac{\nu}{2}-1}e^{-t}{\rm d}t}{\left(\frac{\nu}{2}-1\right)!}

If \nu=2, and solving the integral we have:

F(x; \nu=2) = \dfrac{\int_{0}^{x/2} t^{\frac{2}{2}-1}e^{-t}{\rm d}t}{\left(\frac{2}{2}-1\right)!} = \int_{0}^{x/2} e^{-t}{\rm d}t = 1-e^{-x/2}

In other words, a \chi^{2} distribution with \nu=2 is equivalent to an exponential distribution with rate parameter \lambda=1/2.

The sum of chi-squared is also chi-squared

The moment-generating function (mgf) of a sum of independent variables is the product of the mgfs of the respective variables. The mgf of a \chi^{2}_{\nu} is:

M(t) = (1-2t)^{-\nu/2}

The mgf of the sum of k independent variables that follow each a \chi^{2}_{2} distribution is then given by:

M_{\text{sum}}(t) = \prod_{i=1}^{k} (1-2t)^{-2/2} = (1-2t)^{-k}

which also defines a \chi^{2} distribution, however with degrees of freedom \nu=2k.

Assembling the pieces

With these facts in mind, how to transform the product \prod_{i} P_{i} into a p-value that is uniformly distributed when the global null is true? The product can be converted into a sum by taking the logarithm. And as shown above, the logarithm of uniformly distributed variables follows an exponential distribution with rate parameter \lambda=1. Multiplication of each \ln(P_{i}) by 2 changes the rate parameter to \lambda=1/2 and makes this distribution equivalent to a \chi^{2} distribution with degrees of freedom \nu=2. The sum of k of these logarithms also follow a \chi^2 distribution, now with \nu=2k degrees of freedom, i.e., \chi^{2}_{2k}.

The statistic for the Fisher method is, therefore, computed as:

X = -2 \sum_{i=1}^{k} \ln(P_{i})

with X following a \chi^{2}_{2k} distribution, from which a p-value for the global hypothesis can be easily obtained.

Reference

The details above are not in the book, presumably omitted by Fisher as the knowledge of these derivation details would be of little practical use. Nonetheless, the reference for the book is:

See also

The Fisher’s method to combine p-values is one of the most powerful combining functions that can be used for Non-Parametric Combination.

13 thoughts on “The logic of the Fisher method to combine P-values

    • Thanks for commenting and sorry that the article has confused you. F(x) is a cdf (it’s stated just under the heading “The logarithm of uniform is exponential”).

  1. Pingback: Mission Measurement | The Platinum Standard

  2. Pingback: Bayes Theorem 203: Why Bayes Rules | SINMANTYX

    • Hi Rebecca,

      The combined statistic follows a Chi^2 distribution of which only the right tail is interesting.

      That said, if the p-values for the partial tests (before combination) that are being combined are two-tailed, the combined p-value is automatically represents two-tailed tests. If the p-values before combination are one-tailed, then the combined p-value represents these one-sided tests.

      Please, see Figure 3 of our recent paper: http://onlinelibrary.wiley.com/doi/10.1002/hbm.23115/epdf (it’s Open Access)

      All the best,

      Anderson

      • Hi Anderson – thanks for the earlier reply. In back again to ask another question since the fisher combined prob test is back on my desk. Could one use a different test to look at these pvalues – say the KS test and test for uniformity (Null: pvalues are uniformly distributed, Alt: pvalues are not uniformly distributed) – would this be an equivalent test or have I messed up the logic. I expect it could give a slightly different answer.

        • Hi Rebecca,

          It is possible to compare distributions (as in the KS test) but that would have a different hypothesis. Instead, one would look directly into the Chi^2 distribution to compute the p-values. In many programming languages the cdf of the Chi^2 is accessible readily (e.g., in Matlab/Octave or R). If not, and if for only a couple of tests, those tables in the final pages of old statistic books also help.

          I wouldn’t replace a direct look into the Chi^2 for a test of distributions. If there is a chance that the distribution is not Chi^2 for whatever reason, and if the original data is available, maybe run a permutation test then.

          All the best,

          Anderson

  3. Thank you for this great post. I recently came across the Fisher method for combining p-values and your post helped explain what exactly this does. However, I am still a bit confused as to the rationale for using this method. From what I can tell, this answers the question: what is the probability of obtaining this particular distribution of p-values (or more extreme) given that there is no effect. However, as you state, this is not a global test of significance, and it leaves out a lot of necessary information on sample and effect sizes- so why do it? Can you help either explain its utility- or point to some resources explaining its utility? Thank you!

    • Dear David,

      Thanks for the comments. The test is in fact a global one, i.e., it seeks evidence for a global effect, that may affect very strongly only a few of the individual tests (called “partial tests”), or may affect modestly many of them.

      Perhaps the most interesting part is that the product of p-values isn’t on its own right a p-value, and therefore can’t be used as the p-value for a global test. Instead, this product (or sum of their logs) can be used as the test statistic; once the distribution of this statistic is known, then the p-value for the global test can be obtained, and inference be made.

      One would be interested in using the Fisher’s method when there are multiple independent partial tests, each testing a different null hypothesis, or when the null hypotheses are all the same, but the data collected and used for each partial test are different. It is a type of meta-analysis, in which information from multiple studies are collated together in a single result that summarises the evidence available.

      Hope this helps.

      All the best,

      Anderson

  4. Thank you for your clear and helpful explanation.In this method of meta-analysis,could I use FDR p-value for combining p-value or I should use originally p-values?

    • Hi Elham,
      Should use the original p-values. The FDR p-values are not uniformly distributed under the null and therefore, the Fisher’s statistic would no longer follow a Chi^2 distribution.
      Hope this helps.
      All the best,
      Anderson

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s