A simulation runs 250 tests when the null is wrong: 31 very strong, 64 strong, 40 some, 115 not enough evidence. Estimate the power as a decimal.
Power
The chance a test catches a false null: estimating it by simulation, and how the true effect size, the sample size and the direction of change it.
Key ideas
- Power
The long-run proportion of repeated tests that give (strong or very strong evidence) when the null is wrong and the alternative points the right way.
- Estimating power by simulation
Simulate many samples from the true situation, run the test on each, and count the strong plus very strong results over the total.
- True effect size
How wrong the null is. For a fixed , a larger true effect gives more power.
- Sample size
For a fixed true effect, a larger shrinks sampling variability and raises power. A small effect needs a large sample to reach the same power.
- Direction
A one-sided test pointed away from the truth has almost no power.
- No true effect
Power is only defined when the null is wrong. If the true effect is 0, larger samples do not help: any strong evidence would be a chance result.
- Planning
Researchers choose before collecting data so the test has reasonable power (often 80%) for the smallest effect they care about.
Seed germination
A seed company claims an 80% germination rate. Suppose the true rate is 74% (true effect size ). A simulation runs a left-tailed z-test (, ) on 400 random samples of 60 seeds. Results: very strong 58, strong 106, some 70, not enough 166. Estimate and interpret the power.
Count only strong and very strong: .
Power estimate: .
Interpret: when the company's claim of 80% is wrong and the true rate is 74%, a test based on 60 seeds detects the mistake in only about 41% of repeated samples.
What would raise it: more seeds per sample, or a larger true shortfall. Including the "some evidence" bin would overstate power.
Power ≈ 0.41: this design misses a true 6-point shortfall more often than it catches it.
Check yourself
A planned two-sided t-test of with n = 40 has modest power when the true mean is 53. Select every change that would raise its power.
The true proportion is 0.30. Three analysts test with samples of the same size: Analyst L uses , Analyst T uses , and Analyst R uses . Rank their power from highest to lowest.
Quantity A: power to detect a true when testing with . Quantity B: power to detect a true with the same null and .
A study reports power 0.38. Which interpretation is correct?
A pilot study with 15 participants finds p = 0.40. Select every statement that is justified.
In the seed-germination simulation, the true rate stays 74% but each simulated test now uses 240 seeds instead of 60. The estimated power will most likely: