- Explain why variability must be considered when comparing group means.
- Distinguish null and alternative hypotheses, Type I and Type II errors, and statistical power.
- Interpret a P-value correctly and compare it with a prespecified alpha level.
- Describe how sample size influences standard error, P-values, power, and practical interpretation.
- Connect confidence intervals with hypothesis tests.
- Contrast frequentist P-values with Bayesian posterior probabilities.
- Differentiate superiority, non-inferiority, and equivalence trials.
- Explain why multiple comparisons may require adjustment.
From Variable Data to Statistical Inference
Variability creates uncertainty, and inference uses sample evidence to reason about a population.
Hypothesis testing provides a framework for making data-based decisions. The ideas in this lesson build from variability and inference toward several common forms of clinical-trial testing.
Suppose higher outcomes are better and the new treatment has a larger mean than the old treatment. That fact alone is insufficient: the spread of the individual observations also matters.
If every new-treatment observation exceeds every old-treatment observation, the advantage is clear. As the groups spread out and overlap, some new-treatment outcomes may be below old-treatment outcomes, making the conclusion uncertain.
Hypotheses, Decision Errors, and Power
A test compares two hypotheses and can lead to a correct decision or one of two errors.
A typical hypothesis test begins with two competing statements.
- Null hypothesis, H₀: there is no effect or difference. For a medication and placebo, this says that their effectiveness is equal and their difference is 0.
- Alternative hypothesis, H₁ or Hₐ: an effect or difference exists. In the medication example, the two effects differ.
The null is formulated with the aim of determining whether the evidence is strong enough to reject it in favor of the alternative. The possible study conclusions are to reject H₀ or to fail to reject H₀; the true state remains unknown.
Why is the no-difference claim used as the hypothesis to test?
It supplies a defined condition under which the probability of the observed data can be calculated.
| Study conclusion | H₀ true: no difference | H₀ false: difference exists |
|---|---|---|
| Reject H₀ | Type I error: false positive | Correct conclusion: true positive |
| Fail to reject H₀ | Correct conclusion: true negative | Type II error: false negative |
- Type I error: rejecting H₀ and claiming a difference when no difference exists. Its probability is α.
- Type II error: failing to reject H₀ when a real difference exists. Its probability is β.
- Power: 1 − β, the probability of detecting an effect that truly exists. Greater power is desirable.
Different Priorities for Alpha and Power
A regulatory agency such as the FDA emphasizes α because it seeks to avoid approving an ineffective medication. A common choice is α = 0.05 for a two-sided test. A pharmaceutical company is especially concerned with power because missing a genuinely effective treatment can follow years of research and billions of dollars in spending.
Power commonly ranges from 0.6 to 0.8 in early-phase trials and from 0.8 to 0.95 in later phases.
The Alpha–Beta Trade-off
With sample size held constant, reducing α makes a difference harder to claim. That also increases β because a genuine difference will be missed more often, so power decreases. Raising α has the opposite pattern. Altering sample size changes this trade-off and can reduce both error probabilities.
Understanding the P-Value Through Coin Tosses
The coin experiment shows what a P-value measures and what it cannot establish.
Consider a magician’s kit containing one regular coin, with heads and tails, and one coin with heads on both sides. One coin is selected without revealing which one it is, and repeated tosses provide the evidence.
- H₀: the selected coin is regular, so = 0.5.
- Hₐ: the selected coin has two heads, so = 1.
The evidence accumulates as heads appear repeatedly. Under H₀, one head has probability 0.5, two consecutive heads have probability 0.25, three have probability 0.125, and four have probability 0.0625.
For seven consecutive heads from a regular coin, the probability is:
Thus the P-value is about 0.008: if the coin were regular, seven heads would occur roughly 8 times per 1,000 such sequences. The observed tosses cannot be changed, so this very unlikely outcome leads us to question and reject H₀. The evidence is strong, but it does not prove that the coin has two heads.
Compare the P-value with the alpha level chosen before examining the result. Reject H₀ when P ≤ α, commonly with α = 0.05. When P > α, do not reject H₀ because an outcome this extreme remains sufficiently plausible under the null.
If observation stopped after four heads, would P = 0.0625 justify rejecting H₀ at α = 0.05? Explain.
No. Since 0.0625 is greater than 0.05, H₀ is not rejected.
How Sample Size Changes Statistical Significance
The same observed effect may yield different P-values when sample size changes.
Imagine a randomized trial whose observed treatment difference is 3 units. Even when H₀ states that the true difference is 0, repeated trials would produce a distribution of sample differences. Its standard error depends on sample size.
- Small sample: the null distribution is wide and the two-sided P-value for a difference of 3 is 0.32, so H₀ is not rejected.
- Moderate sample: the standard error is smaller, the distribution is narrower, and the same difference gives P = 0.134. This remains above 0.05.
- Large sample: the curve becomes tighter still, and the unchanged difference produces P = 0.0026, a statistically significant result.
As sample size rises, the standard error falls because sample size appears under a square root in the denominator. Consequently, even a very small effect can become statistically significant in a sufficiently large sample.
Power analysis begins by asking an investigator which effect would be clinically meaningful—for example, 3 units. It then identifies the smallest affordable sample that provides high power, such as 0.8, to detect that effect if it exists. A study should not be so small that meaningful effects are missed or unnecessarily huge merely to make tiny effects significant.
Reading Confidence Intervals Alongside Tests
Confidence intervals show effect magnitude and precision while connecting directly to significance tests.
For example, an estimated treatment difference of 3 with a 95% confidence interval of 3 ± 2 gives the interval (1, 5). In everyday wording, we are 95% confident that the true difference is within this range.
In frequentist terms, the true value is fixed while intervals vary from study to study. If the study were repeated many times, about 95% of the calculated 95% intervals would cover the true value.
For a difference whose null value is 0, a 95% interval that excludes 0 corresponds to statistical significance at the 5% level. If the interval contains 0, the result is not significant at that level. Unlike a lone P-value, the interval also displays the estimated effect’s size and precision.
Bayesian Updating as an Alternative Framework
Bayesian analysis updates prior beliefs with observed data to obtain posterior probabilities.
Classical testing uses quantities such as P-values and α. Bayesian analysis offers another approach that has become more widely used, including by the FDA, as computing has made its calculations easier to perform.
Bayesian reasoning starts with a prior belief, combines that belief with experimental data through Bayes’ rule, and produces a posterior belief. That posterior can serve as the prior for the next set of observations, allowing repeated updating.
Return to the coin experiment. Before any toss, equipoise assigns probability 0.5 to the regular coin and 0.5 to the two-headed coin.
- After one head, the posterior probability of the regular coin falls while the probability of the two-headed coin rises.
- After a second head, the probabilities are updated again in the same directions.
- After seven heads, the posterior probability is about 0.8% for the regular coin and 99.2% for the two-headed coin.
These Bayesian values make claims about the hypotheses themselves. By contrast, the frequentist P-value of 0.008 says only how likely seven heads would be if the coin were regular.
Sequential trials can repeatedly update posterior beliefs as new data arrive and stop once the evidence is sufficient to conclude efficacy or futility.
Three Trial Goals and Their Margins
Superiority, non-inferiority, and equivalence trials pose different questions and therefore use different hypotheses.
Some trials ask whether an experimental treatment is better than a control. Others ask whether it avoids an unacceptable loss of effectiveness or whether two treatments are sufficiently similar. Non-inferiority and equivalence require a margin δ.
Testing for Superiority
- Clinical aim: show that the experimental treatment is more effective than the control.
- Statistical hypotheses: H₀: experimental = control, so the difference is 0; Hₐ: experimental ≠ control, so the difference is not 0. The two-sided alternative allows the experimental treatment to be either better or worse.
- Conclusion: a 95% confidence interval wholly above 0 supports superiority; one wholly below 0 supports inferiority; an interval containing 0 does not justify rejecting H₀.
Testing for Non-Inferiority
- Clinical aim: establish that the experimental treatment is not worse than the control by more than δ.
- Statistical hypotheses: H₀: experimental ≤ control − δ; Hₐ: experimental > control − δ.
- Margin δ: the largest acceptable reduction in efficacy, selected by subject-matter experts rather than statisticians.
- Conclusion: claim non-inferiority when the entire 95% interval is right of −δ. If it is also wholly above 0, it supports both non-inferiority and superiority. If it crosses 0 but remains right of −δ, only non-inferiority is supported. An interval that covers −δ is inconclusive.
Testing for Equivalence
- Clinical aim: demonstrate that two treatments differ by less than the symmetric margins ±δ.
- Statistical hypotheses: H₀: |experimental − control| ≥ δ; Hₐ: |experimental − control| < δ.
- Conclusion: claim equivalence only when the complete 95% interval lies between −δ and +δ. If the interval reaches beyond either margin, equivalence cannot be claimed.
Why does choosing δ require careful subject-matter judgment?
An overly large margin can label meaningfully different treatments as equivalent or non-inferior, while an overly small margin can make similarity extremely difficult to demonstrate.
Controlling Error Across Multiple Tests
Running many tests creates more opportunities for false-positive findings.
Testing numerous treatments, endpoints, subgroups, or analysis choices increases the opportunity to obtain at least one Type I error. Repeated testing can eventually produce a result below 0.05 even when no genuine difference exists.
Think of one dart player who needs 20 throws to reach the bullseye and another who hits it on the first throw. A success found after many opportunities is not interpreted like a success from a single opportunity. Statistical testing similarly requires attention to how many chances were available to find significance.
Situations Requiring Formal Adjustment
- Use formal adjustment when a study has more than one primary endpoint or primary hypothesis, such as two outcomes judged equally important.
- Use formal adjustment for several treatment comparisons, such as A versus placebo, B versus placebo, and A versus B.
Making the Threshold More Stringent
The Bonferroni correction is a simple, conservative formal method: divide the overall α by the number of tests. Two tests use 0.05/2 = 0.025, while three use 0.05/3 ≈ 0.0167. This makes rejection harder and protects the overall Type I error rate.
Exploratory secondary or subgroup analyses need not always receive the same formal treatment as primary analyses, but their P-values still require caution. Investigators may use a stricter informal cutoff such as 0.01 or explicitly state that many tests were performed and the findings should not be overinterpreted.
Two Ways to Manufacture Significance
- Use an extremely large sample. Even a tiny effect can become statistically significant, although its clinical value remains a separate issue.
- Try many outcomes, subgroups, or analyses until one gives P < 0.05. That finding may simply reflect repeated opportunities for chance, which multiplicity adjustment is intended to address.
Common misconceptions
Watch out: A P-value of 0.008 means there is a 0.8% probability that H₀ is true.
Instead: It means that data at least this extreme have probability about 0.008 under H₀; it does not assign a probability to H₀ itself.
Watch out: If P = 0.008, then the alternative hypothesis has probability 99.2%.
Instead: That conclusion is not available from a frequentist P-value. A Bayesian posterior probability can concern a hypothesis, but it is calculated and interpreted differently.
Watch out: A P-value of 0.008 is the probability that chance caused the result.
Instead: The value is the probability of observing an equally or more extreme result if the null hypothesis were true.
Watch out: Any statistically significant effect must also be clinically important.
Instead: A large sample can make a very small effect statistically significant. Clinical experts must separately judge whether its magnitude matters in practice.
Lesson recap
- Statistical inference moves from sample results to conclusions about a population.
- H₀ commonly represents no difference and is set up so the evidence can be assessed for possible rejection.
- α is the chance of a Type I error, β is the chance of a Type II error, and power is 1 − β.
- A P-value measures how probable equally or more extreme data would be under H₀; it is not the probability that H₀ is true.
- P ≤ α is statistically significant, but statistical significance alone does not establish clinical importance.
- Confidence intervals display effect size and precision; for a difference, excluding 0 corresponds to significance at the matching level.
- Larger samples reduce standard error, increase power, and make smaller effects easier to detect.
- Bayesian inference combines prior belief with data to obtain posterior probabilities, which are distinct from P-values.
- Superiority, non-inferiority, and equivalence designs use different hypotheses and confidence-interval criteria.
- Several primary tests require multiplicity control, with Bonferroni adjustment providing one conservative method.
Together, these ideas provide a foundation for interpreting statistical evidence, recognizing design trade-offs, and separating numerical significance from practical meaning.