How GO::TermFinder calculates P-values
The GO::TermFinder attempts
to determine whether an observed level of annotation for a group of genes is
significant within the context of annotation for all genes within the
genome. Suppose that we have a total
population of N genes, in which M have a particular annotation. If we observe x genes with that
annotation, in a sample of n genes, then we can calculate the probability of
that observation, using the hypergeometric distribution (e.g., see http://mathworld.wolfram.com/HypergeometricDistribution.html)
as:
where generically,
which is the number of
permutations by which r entities can be selected from n entities, is calculated
as:
To actually generate a
p-value, rather than a simple probability, instead of asking the
question, what is the probability of having 5 out of 10 genes with
this annotation, given that 42 out 6000 have it,
we ask the question what is the probability of having 5 or
more out of 10 genes having
this annotation. This is what a p-value
is Ð the chance of seeing your observation, or better, given the background
distribution. We
calculate this by summing our probabilities for 5 out of 10, 6 out of 10, 7 out of 10 etc. Thus the probability of seeing x or
more genes with an annotation, out n, given that M in the population of N have
that annotation, is:
Note that this is the same
as saying whatÕs the chance of getting at least x successes, and can also be represented by:
Typically, a cut-off for
p-values, known as the alpha level, is chosen, such that p-values below the
alpha level are deemed significant.
The alpha level is the chance taken by
researchers to make a type one error.
The type one error is the error of incorrectly declaring a difference,
effect or relationship to be true due to chance producing a particular state of
events. Customarily the alpha
level is set at 0.05, or, in no more than one in twenty statistical tests the
test will show 'something' while in fact there is nothing. In the case of more than one
statistical test the chance of finding at least one test statistically significant
due to chance fluctuation, and to incorrectly declare a difference or
relationship to be true, increases.
In five tests the chance of finding at least one difference or
relationship significant due to chance fluctuation equals 0.22, or one in five. In ten tests this chance increases to
0.40, which is about one in two.
Thus we need to make an adjustment that will correct for multiple
hypotheses. The Bonferroni method
adjusts the alpha level of each individual test downwards to ensure that the
overall risk for a number of tests remains 0.05. Even if more than one test is done the risk of finding a
difference or effect incorrectly significant continues to be 0.05. To do this, it simply divides the
alpha-level by the number hypotheses that were tested, so if 20 hypotheses were
tested, then instead of using an alpha-level of 0.05, an alpha level of 0.0025
would be used. Alternatively, the
p-values can be adjusted, by multiplying by the number of hypotheses that were
tested, and the alpha-level can be kept the same. This approach is the one that GO::TermFinder takes. In the case of GO::TermFinder, the
value used for the Bonferroni correction is the number of nodes to which the
genes of interest are collectively annotated, excluding those nodes which only
have a single annotation in the background distribution, which a priori cannot be significantly enriched. The Bonferroni correction assumes
however that all hypotheses are independent. In the case of the GO::TermFinder, each hypothesis is a node
in the Gene Ontology, which has two or more annotations (either directly or
indirectly) from the tested group of genes (nodes with only one annotation are
not tested). Because these
hypotheses form a Directed Acyclic Graph (which is a subgraph of the full GO
DAG) there are thus relationships between the hypotheses, meaning that they are
not independent, and thus the Bonferroni correction may not be appropriate.
GO::TermFinder
also includes a mode for correcting multiple hypotheses by running 1000 simulations. The corrected p-value is calculated as
the fraction of simulations having any p-value as good or better than the
observed p-value. Comparison of
simulation corrected p-values with Bonferroni corrected p-values actually
suggests that the Bonferroni correction is not conservative enough.
GO::TermFinder
also calculate a False Discovery Rate, as a mean of sidestepping the issues of
p-values and multiple hypotheses.
Classic Multiple hypothesis correction can be very conservative, as it
tries to maintain the probability of getting any false positives at a particular alpha
level. The False Discovery Rate
instead allows a user to choose a cut off that has an acceptable level of false
discovery. Below is a example data
generated from GO::TermFinder that allows comparison of corrected p-values and
False Discovery Rate.
Table 1. Comparison
of Bonferroni corrected p-values, simulation corrected p-values, and False
Discovery Rate for the 28 most significant GO nodes, for a group of genes that
show sensitivity to 1M NaCl and 10mM nystatin (Giaever et al, 2002; SNF7 STP22 VPS28 SNF8
VPS36 VPS25 YGR122W RIM20 RIM21 RIM8 RIM101 DFG16 RIM9 YGL046W RIM13
YNR029). Note that the Bonferroni
correction is up to 2.8 fold less
conservative than the simulation method that controls the Family Wise Error
Rate. N/A Ð Not applicable Ð cases
where no p-values better than that nodeÕs p-value were seen in simulations.
