A one-sample t-test with gives . What is the estimated effect size? Give the exact value.
Estimated effect size
Why a p-value cannot tell you how wrong the null is, how measures it in SD units, the course's small/moderate/large scale, and comparing studies of different sizes.
Key ideas
- First limitation of tests
A p-value measures the strength of evidence against , not the size of the error made. Statistically significant does not mean practically important.
- Estimated effect size
The gap between the statistic and the null value in units of the data's own spread (individual-level SD), not the standard error. Unitless.
- Course scale
small; moderate; large. Printed on the Exam 2 help card; other books use 0.2/0.5/0.8, so use these.
- Independent of n
Collecting more data does not make an effect bigger. Even the two-group shortcut, which has and in it, estimates a quantity that does not depend on sample size.
- Same sign as the statistic
always has the sign of z or t.
- Replication
To judge whether a follow-up study with a different replicates an earlier one, compare the effect sizes, not the p-values.
- Order of analysis
Start with the test. If the p-value is small, ask whether the effect is large enough to matter. A tiny effect applied at a huge scale can still matter operationally.
Formulas (all three are on the Exam 2 help card)
Two studies of a reading app
Two school districts test whether a reading app changes students' mean score gain from 0 (two-sided one-sample t-tests). District A uses 400 students and gets , . District B uses 25 students and gets , . Compare the evidence and the effect sizes.
Evidence: A's p-value is very strong evidence of a nonzero mean gain; B's 0.057 is some evidence.
Effect sizes: A: (small). B: (moderate).
Why they disagree: A's huge sample turns a small effect into a large t. B's estimated effect is twice as large, but 25 students leave a lot of uncertainty.
Write-up for A: very strong evidence that the true mean gain for the district's students is not 0, but the estimated effect is small, so the improvement may be modest in practical terms.
A has stronger evidence but a smaller estimated effect (0.20 versus 0.40). Smaller p-value does not mean bigger effect.
Check yourself
A study of 9 people gives and a two-sided p-value of about 0.30. A colleague says: "There is no effect." Which response is best?
An original study found . Three follow-ups with different sample sizes report: Study 1, , , ; Study 2, , , ; Study 3, , , . Which follow-up best replicates the original finding?
Select every true statement about the estimated effect size .
A two-sample t-test with 50 people in each group gives . Using the help card shortcut, what is ? Give the exact value.
A national survey of 50,000 adults tests whether mean weekly screen time changed from last year: , . Which summary is best?