MaizePrepSTATS 250
Exam 2 · Lec 08

Estimated effect size

Why a p-value cannot tell you how wrong the null is, how d^\hat d measures it in SD units, the course's small/moderate/large scale, and comparing studies of different sizes.

30 min

Key ideas

First limitation of tests

A p-value measures the strength of evidence against H0H_0, not the size of the error H0H_0 made. Statistically significant does not mean practically important.

Estimated effect size d^\hat d

The gap between the statistic and the null value in units of the data's own spread (individual-level SD), not the standard error. Unitless.

Course scale

∣d^∣≤0.25|\hat d| \le 0.25 small; 0.25<∣d^∣≤0.750.25 < |\hat d| \le 0.75 moderate; ∣d^∣>0.75|\hat d| > 0.75 large. Printed on the Exam 2 help card; other books use 0.2/0.5/0.8, so use these.

Independent of n

Collecting more data does not make an effect bigger. Even the two-group shortcut, which has n1n_1 and n2n_2 in it, estimates a quantity that does not depend on sample size.

Same sign as the statistic

d^\hat d always has the sign of z or t.

Replication

To judge whether a follow-up study with a different nn replicates an earlier one, compare the effect sizes, not the p-values.

Order of analysis

Start with the test. If the p-value is small, ask whether the effect is large enough to matter. A tiny effect applied at a huge scale can still matter operationally.

Formulas (all three are on the Exam 2 help card)

One proportion
d^=π^−π0π0(1−π0)=zn\hat d=\frac{\hat\pi-\pi_0}{\sqrt{\pi_0(1-\pi_0)}}=\frac{z}{\sqrt n}
One mean
d^=μ^−μ0σ^=tn\hat d=\frac{\hat\mu-\mu_0}{\hat\sigma}=\frac{t}{\sqrt n}
Divide by σ^\hat\sigma, not by the SE.
Two means
d^=tn1n2n1+n2\hat d=\frac{t}{\sqrt{\dfrac{n_1n_2}{n_1+n_2}}}
The connection
test statistic=d^×n\text{test statistic}=\hat d\times\sqrt n
A big statistic can come from a big effect, a big sample, or both.
Worked example

Two studies of a reading app

Two school districts test whether a reading app changes students' mean score gain from 0 (two-sided one-sample t-tests). District A uses 400 students and gets t=4.0t = 4.0, p≈0.000075p \approx 0.000075. District B uses 25 students and gets t=2.0t = 2.0, p≈0.057p \approx 0.057. Compare the evidence and the effect sizes.

  1. Evidence: A's p-value is very strong evidence of a nonzero mean gain; B's 0.057 is some evidence.

  2. Effect sizes: A: d^=4.0/400=0.20\hat d = 4.0/\sqrt{400} = 0.20 (small). B: d^=2.0/25=0.40\hat d = 2.0/\sqrt{25} = 0.40 (moderate).

  3. Why they disagree: A's huge sample turns a small effect into a large t. B's estimated effect is twice as large, but 25 students leave a lot of uncertainty.

  4. Write-up for A: very strong evidence that the true mean gain for the district's students is not 0, but the estimated effect is small, so the improvement may be modest in practical terms.

Answer

A has stronger evidence but a smaller estimated effect (0.20 versus 0.40). Smaller p-value does not mean bigger effect.

Check yourself

1
Lec 08

A one-sample t-test with n=144n = 144 gives t=3.0t = 3.0. What is the estimated effect size? Give the exact value.

Math shortcuts are on for STATS 250. Type ^ then digits for powers, _ then digits for subscripts.
2
Lec 08

A study of 9 people gives d^=0.37\hat d = 0.37 and a two-sided p-value of about 0.30. A colleague says: "There is no effect." Which response is best?

3
Lec 08

An original study found d^=0.48\hat d = 0.48. Three follow-ups with different sample sizes report: Study 1, t=3.9t = 3.9, p=0.0002p = 0.0002, d^=0.21\hat d = 0.21; Study 2, t=1.6t = 1.6, p=0.12p = 0.12, d^=0.46\hat d = 0.46; Study 3, t=2.5t = 2.5, p=0.015p = 0.015, d^=0.90\hat d = 0.90. Which follow-up best replicates the original finding?

4
Lec 08

Select every true statement about the estimated effect size d^\hat d.

5
Lec 08

A two-sample t-test with 50 people in each group gives t=3.0t = 3.0. Using the help card shortcut, what is d^\hat d? Give the exact value.

Math shortcuts are on for STATS 250. Type ^ then digits for powers, _ then digits for subscripts.
6
Lec 08

A national survey of 50,000 adults tests whether mean weekly screen time changed from last year: p=0.002p = 0.002, d^=0.03\hat d = 0.03. Which summary is best?

MaizePrep is an independent study resource. It is not affiliated with, sponsored by, or endorsed by the University of Michigan. Course names and numbers are used only to describe which class the material covers. Every practice question is original, written in the style of the course.