Power and sample size #

A study can miss an effect that is there. Not through any mistake: with too few cases, a real difference sits inside the noise, the test cannot tell it from nothing, and the report says “no significant effect” of something that exists. How often that happens is a number, it can be computed before a single case is collected, and computing it is what the analysis planner is for. This page explains power and the quantities it trades against, the three questions a power analysis can answer, what moves the answer, planning for precision instead of detection, and the two practical corrections – unequal groups and drop-out. The numbers come from the café data, prepared as that page describes, and every one of them can be reproduced with the planner or the module named beside it.

Four quantities, one equation #

Power is the probability that a test finds an effect of a given size when the effect is real: the chance that, if the dependent variable really does differ between the levels of the grouping variable by the amount you assume, the test comes out significant. It is the complement of the Type II error rate – a power of 0.80 is a 20% chance of missing the effect – and the convention is to aim for 0.80, or 0.90 when a miss is expensive. The café’s two headline results show both ends. The branch comparison of spend_before – 101 Riverside members against 99 Station members, a gap of a little under half a standard deviation (d = 0.45) – had a power of 0.89 for an effect that size: nine such studies in ten would have found it. The change in visits after the menu – 200 members, each their own control, a change of a seventh of a standard deviation (dz = 0.14) – had a power of 0.50: a coin toss, which is the honest reading of its p = 0.051. A nurse planning a two-ward comparison (ward → grouping variable, days to recovery → dependent variable) and a teacher planning a two-class one (class → grouping, test score → dependent) ask the same question in the same words: how likely is my design to see the difference I expect, if it is there?

Power (1 − β) is the same quantity written as the planner’s input label: β is the Type II error rate, the probability of a miss, and 1 − β the probability of a hit. The default of 0.80 is the convention; raising it to 0.90 for the café’s branch comparison lifts the required sample from 79 to 105 per group, and to 0.95, 130.

Power analysis is the calculation that ties four quantities together – the size of the effect, the significance level, the sample size and the power – so that fixing any three gives the fourth. A larger effect, a looser α, more cases or a lower target power each make the others cheaper, and the planner’s Solve for control names which of the three usual unknowns you are after: the sample size for a planned study, the power of a design you already have, or the smallest effect a design can find. The fourth quantity, α, is fixed by convention and entered, not solved for.

Expected effect size is the one input a power analysis cannot supply for itself, and the one the answer is most sensitive to. It is the effect you assume in order to plan – not the effect you hope for, and not the one that would be convenient – and it has three honest sources: a meta-analysis or prior study of the same kind of effect, a pilot, or the smallest effect you would care about. The café’s branch gap of d = 0.45 came from an earlier season’s data; a plan built on it asks for 79 members per branch. Had the planner been left at its default of 0.50 the answer would have been 64, and a study of 64 per branch would have had a power of 0.71 for the effect that was actually there. The effect sizes page explains each measure; the planner’s effect size converter turns the one a paper reports into the one a test plans with.

Smallest effect size of interest is the effect below which you would not care whether the null was true or not – the alternative to guessing what the effect will be. If a menu change that raised spend by less than 0.50 per visit would not pay for itself, plan to detect 0.50 ⁄ 3.63 = 0.14 standard deviations and no smaller; a study powered for that is powered for everything that matters, and a non-significant result from it means “if there is an effect, it is too small to matter”, which a non-significant result from an underpowered study never means. It is the most defensible planning input there is, because it comes from the question rather than from a guess.

Significance level (α) is the significance level the planned test will use, and in a power analysis it is a cost: the stricter the bar, the more cases it takes to clear it with the same effect. The café’s branch comparison needs 79 per group at α = 0.05 and 118 at α = 0.01; with 100 per group, its power for d = 0.45 is 0.89 at 0.05 and 0.72 at 0.01. Enter the level the report will actually use – after any multiple comparison adjustment, a family of six tests under Bonferroni runs each at 0.0083, and the plan should say so.

Required sample size is the number of cases at which the planned test reaches the target power for the expected effect, and the usual reason to open the planner. For an independent-samples t-test at d = 0.45, α = 0.05 two-tailed and a power of 0.80, it is 79 per group, 158 in all – the café’s 200 members were more than enough for the branch comparison. For the change in visits, dz = 0.14 on a paired test, it is 403 pairs: twice what the café had, which is why the study found p = 0.051 and not less. The number is always rounded up, and it is a floor, not a target – the attrition section says what to add.

Sample size (per group) is the required sample size as the planner reports it for a two-group test: cases in each group, with the total in brackets. 79 per group is 158 members.

Number of pairs is the sample size of a paired design – members measured twice, or matched couples – counted as pairs, since a pair is the unit the test works on. 403 pairs is 403 members and 806 measurements.

Total sample size is the sample size of a test that has no groups to count by – a correlation, a regression – given as the number of cases in the whole file. A correlation of 0.24 needs 134 cases for a power of 0.80; the café’s 192 members with both overall and spend_after gave 0.92.

Underpowered study is one whose power for the effect it was looking for is well below 0.80. Its non-significant results say almost nothing – “no significant change in visits” from a design with a power of 0.50 is a report on the design, not on the visits – and its significant results are worse than they look, as the last section explains. The remedy is decided before the data are collected, which is the whole point of planning.

Three questions the planner answers #

A priori power analysis is the planning use: the effect size, α and target power are set and the required sample size is solved for, before the data exist. It is the only kind of power analysis that changes what a study does, and the only kind a reviewer asks for. Every number in the paragraphs above was one.

Achieved power is the planner’s answer when the sample size is given instead: the power a design of that size has for the expected effect. It is the question to ask of a study whose size is fixed by circumstance – a class of 30, a ward of 45 beds, a survey that already ran – and the café’s own results are the worked instance: 101 and 99 members give a power of 0.89 for d = 0.45, and 200 pairs give 0.50 for dz = 0.14. Enter the sample you will have and the effect you expect from the literature; the answer is honest either way, and a low one is the reason to change the design or the question.

Minimum detectable effect is the third solve: the smallest effect a given design finds with the target power. With 100 members per branch and a power of 0.80 it is d = 0.40 – so the café’s branch design could reliably detect a gap of two fifths of a standard deviation, and had a power of only 0.42 for one of a quarter. With 30 per group it is d = 0.74, a large effect by Cohen’s benchmarks, and with 200 pairs on a paired test it is dz = 0.20 – a fifth of a standard deviation, which the real change of 0.14 fell short of. It is the number to quote when a study did not find something: “the design had 80% power to detect a change of 0.20 SD or more” tells the reader what the null result rules out.

Observed power – also post hoc power – is the power computed after the test, with the effect size the study itself observed put in as the expected one. It looks like a diagnosis of a null result and it is not: for a given design it is a one-to-one function of the p-value, so it adds nothing the p already said. A two-tailed result at exactly p = 0.05 always has an observed power of 0.50; the visits change at p = 0.051 has 0.50; any non-significant result has less than a half, by construction. The circularity is what makes it worthless – the observed effect is the estimate whose uncertainty was the problem, and feeding it back in does not reduce that uncertainty. The question that observed power seems to ask is answered by the minimum detectable effect computed for an effect size chosen on other grounds, and by the confidence interval on the effect that was found: the visits change’s interval of −0.003 to 1.28 says what the study left open, and no power figure improves on it.

How sure the answer is #

Power curve is power drawn against sample size for a fixed effect and α – the picture of what a study buys with each additional case. For the branch comparison at d = 0.45 it rises from 0.16 at 10 per group through 0.40 at 30, 0.61 at 50 and 0.80 at 79 to 0.89 at 100, and flattens: 200 per group gives 0.99, and the last hundred members per group bought a tenth. The curve is steepest exactly where a study is most likely to be – around the sample size that just reaches 0.80 – so a plan that lands there and then loses a few cases loses power fast. The planner’s Show power curve draws it, with a crosshair on the calculated result.

Alongside every answer the planner prints a Sensitivity analysis table – the power at half, three quarters, one and a quarter and one and a half times the reference sample size. For the branch comparison’s 79 per group the row reads 40 → 0.51, 60 → 0.69, 79 → 0.80, 99 → 0.88, 119 → 0.93. Read it as the cost of being wrong: a study that recruits three quarters of its planned sample has a power of 0.69, not 0.80. The Show sensitivity heatmap button extends the same idea to two dimensions, power across a grid of sample sizes and effect sizes, and is the honest way to show what happens if the expected effect was optimistic as well as the recruitment. An effect size that turns out smaller than assumed is the commoner miss, and the more expensive: the same 79 per group has a power of 0.80 at d = 0.45 and of 0.24 at d = 0.20.

What moves power #

Power rises with the effect size, the sample size and α, and with the amount of noise a design removes. The café shows each lever.

The effect. Power grows with the sample for any fixed effect, and a larger effect reaches a given power with fewer cases. For 80% power at α = 0.05, a two-group comparison needs 26 per group at d = 0.8, 64 at d = 0.5, 79 at d = 0.45, and 394 at d = 0.2: the small effect costs fifteen times the large one, because the required sample grows with the square of one over the effect.

The design. Measuring the same members twice, as the café’s till columns do, removes the between-member spread from the comparison, and a paired test sees the change against only the spread of the change within a member. The rise in spend after the menu, dz = 0.38 on the paired scale, needs 57 pairs for 80% power. The same 1.09 rise measured on two independent sets of members – one group’s spend before, another’s after – would be d = 0.30 against the full spread of spending, and would need 176 per group, 352 members. The pairing bought a factor of six; the independence section of the assumptions page shows the same numbers from the other side.

The tail. A one-tailed test spends all of α on the predicted direction and so has more power there – 0.94 rather than 0.89 for the branch comparison at 100 per group, or 62 per group rather than 79 for the same power – and none in the other: for a real effect the wrong way it is close to zero. Plan one-tailed only under the conditions the hypothesis testing page sets out, and enter the tail the report will use.

The noise. Anything that shrinks the standard deviation the effect is measured against raises power at no cost in cases: a more reliable questionnaire, a covariate that explains part of the outcome, a homogeneous sample where a heterogeneous one would do. Unreliable measures work the other way – two measures with a reliability of 0.70 each show a true correlation of 0.50 as about 0.35, and the plan must be made for 0.35 – and the planner’s attenuation tool does that arithmetic; the reliability manual explains the coefficient it takes.

Allocation ratio is how a two-group total is split between the groups, and equal groups give the most power for the total. Unequal groups are often forced – the café’s members who would recommend it outnumber those who would not by 135 to 65, and a comparison of their prior visits (d = 0.33) has to take the split it is given. The planner’s group allocation optimizer says what the split costs: at 135 ⁄ 65 the power is 0.585, where 100 ⁄ 100 would give 0.641; at 3 : 1 it would be 0.520. The loss grows with the imbalance and stays small until the smaller group gets small – 2 : 1 costs about five points here, and 9 : 1 (180 against 20) would leave 0.29 – so an imbalance of two or three to one is rarely worth fighting, and power is recovered by enlarging the smaller group rather than the larger: with the 135 recommenders fixed, reaching 0.80 would take 157 non-recommenders.

Equal allocation is the 1 : 1 split, the most powerful use of a fixed total, and the row the optimizer’s table highlights as the reference.

Power loss vs equal allocation is the optimizer’s headline for an unequal split: the difference, in percentage points of power, between the entered allocation and the equal one on the same total – 5.61 points for the café’s 135 ⁄ 65. It is the price of the imbalance, and the number to weigh against the cost of recruiting the smaller group up.

The effect size each test plans with #

The planner asks for the effect in the measure its test uses, and the effect sizes page defines most of them: Cohen’s d for t-tests, dz for paired ones, r for a correlation, Cohen’s w for a chi-square test and Cohen’s h for proportions. Three the planner uses that page does not name:

Cohen’s f is the effect size of a one-way ANOVA – the spread of the group means, in standard-deviation units of the spread within groups – and it is η² rewritten,

so η² of 0.06 is f of 0.25, Cohen’s “medium”. For two groups it is half of d. Splitting the café’s members into three age bands – up to 30, 31 to 45, and 46 and over, a binning rule with cut points at 31 and 46 – gives 47, 102 and 46 members whose spend before the menu averaged 6.85, 9.07 and 9.48: F(2, 192) = 8.42, p < 0.001, η² = 0.08 and f = 0.30. A three-group ANOVA planned for f = 0.30 needs 37 per group, 111 in all, and the café’s 195 members with a valid age gave it a power of 0.97.

Cohen’s f² is the regression effect size, the variance a model explains over the variance it leaves – the f² of the regression basics page – and the planner returns not a sample size but the error degrees of freedom the model needs, from which n is the df plus the predictors plus one. Predicting overall from taste and wait explains 32% of its variance, f² = 0.47, and 21 error df – 24 members – would have found that with 80% power; Cohen’s “small” of 0.02 needs 482 error df, and the same two predictors 485 members. That regression’s 178 members were for precision, not detection.

Cohen’s conventions are the small, medium and large benchmarks the planner shows in its quick reference – 0.20, 0.50 and 0.80 for d; 0.10, 0.30 and 0.50 for r; 0.10, 0.25 and 0.40 for f; 0.02, 0.15 and 0.35 for f²; 0.10, 0.30 and 0.50 for w – and the defaults its effect size fields open at. They are a vocabulary, not a plan: “medium” is a guess dressed as a convention, and a study powered for d = 0.50 in a field where the real effects are 0.20 will miss three quarters of them. The effect sizes page has Cohen’s own caveat; the planning rule is that the expected effect comes from evidence or from the question, and the convention is what to write when there is neither, said in so many words.

On the café’s categorical outcome the benchmarks read as follows. Whether a member would recommend the café differs a little by branch – 72% at Riverside, 63% at Station – which is h = 0.21 for a two-sample proportion test and w = 0.10 for a chi-square test of independence on the same 2 × 2 table, both “small”. A two-sample proportion test planned for h = 0.21 needs 356 per group; the chi-square for w = 0.10 with one degree of freedom needs 785 in all, and the café’s 200 gave it a power of 0.29 – which is why the comparison found nothing, and not evidence that the branches are alike. The 135 ⁄ 65 split of the recommendation itself against an even one is w = 0.35, and a goodness-of-fit test of that needs 65 cases: the café had a power of 0.999 for it.

Precision instead of power #

Not every study is looking for an effect. A survey that wants to know the members’ average spend, the share who would recommend the café or the strength of the link between satisfaction and spending is estimating a quantity, not testing a null, and the sample it needs is set by how precise the estimate has to be – the width of its confidence interval – rather than by any power.

Precision planning is sample size solved from a target interval width: the sample at which the confidence interval for the estimate will be no wider than ± the margin you name. The planner’s precision planning tool takes the estimate type, the confidence level, the expected spread and the margin, and its table shows the sample at half and double the margin beside the one asked for. The relationship is the one the interval’s width obeys: the margin shrinks with the square root of the sample, so halving it takes four times the cases.

Margin of error is the half-width of the confidence interval – the ± after the estimate – and the precision that is being planned for. The café’s 200 members put the mean spend at 8.71 ± 0.51 at 95%; a margin of ± 0.25 would take 813 members, and ± 1.00 only 54. Choose it in the units of the estimate and by what the number is for: a margin of ± 0.50 on a spend of 8.71 is a 6% uncertainty, enough for a business decision, and too loose to detect a change of 0.50.

Expected standard deviation is the spread the precision of a mean depends on, and the one number the planner needs from outside: a pilot, an earlier survey, the literature. The café’s spend_before has an SD of 3.63; entering it with a margin of ± 0.50 gives 205, which is why 200 members produced ± 0.51. An SD guessed too small gives a sample too small, and the estimate arrives wider than planned; when in doubt, plan with the larger of two plausible values.

Expected proportion is the share the precision of a proportion depends on, and here a guess costs little: the interval is widest at 0.50 and narrower towards 0 and 1, so 0.50 is the safe entry when nothing is known. Two thirds of the café’s members would recommend it, 67.5% ± 6.5 on 200; a margin of ± 5 points takes 338 members at 0.675 and 385 planned conservatively at 0.50, and ± 3 points, 937.

Expected correlation is the coefficient the precision of a correlation depends on, because the interval for r narrows as r grows – a strong correlation is pinned down by fewer cases than a weak one. The café’s r = 0.55 between overall and taste has a 95% interval of 0.44 to 0.64 on 193 members; a margin of ± 0.10 around 0.55 takes 190 members and ± 0.05, 751. The weaker r = 0.24 between overall and spend_after is looser for the same effort – ± 0.13 on 192 members, an interval of 0.10 to 0.37 – and a margin of ± 0.10 around it would take 343. The planner searches for the sample rather than solving a formula, and reports a margin that no sample within its limit reaches as Not reachable.

Two corrections before the plan is final #

Attrition is the share of the recruited cases that will not reach the analysis – drop-outs, unusable records, members who skip the question – and the required sample is inflated to allow for it: divide by one minus the expected loss. The 79 per group of the branch comparison becomes 93 at a 15% loss and 99 at 20%; the 403 pairs for the visits change become 504 at 20%. The café’s own file shows where the loss comes from: 15 members left wait blank and 7 left overall blank, so an analysis of their correlation has 178 of 200 – an 11% attrition that no one planned for. Report both numbers – the sample size the analysis needed and the one recruited to reach it.

Pilot study is a small run of the design before the real one, and its honest uses are the SD, the response rate and the feasibility of the procedure – not the effect size. An effect estimated from a pilot is too uncertain to plan on: the first 20 members of each branch give 8.62 against 10.15, d = 0.48 with a 95% interval of −0.15 to 1.11 – anywhere from a small effect the wrong way to a very large one – and a plan built on the point estimate would be built on noise. Use the pilot’s spread with an effect size from the literature or from the smallest effect of interest, and never both the pilot’s estimate and the pilot’s data in the final analysis.

What an underpowered study finds #

A study with a power of 0.50 misses the effect half the time. What is less obvious is what happens the other half: the runs that reach significance are the ones where noise happened to add to the effect, so the effect they report is inflated, and at low power badly so. The café’s visits change makes the point by simulation – 20,000 studies drawn from a population where the true change is dz = 0.14, each tested at α = 0.05, with a seed so the run repeats:

Type M error – M for magnitude – is that inflation: the factor by which a significant result overstates the true effect, on average. At 80% power it is small; at the power of the café’s actual design it is 1.4, and had the café polled 50 members the significant results among the one in six studies that got one would have averaged more than two and a half times the real change. Every underpowered literature is a collection of such results, and a meta-analysis of it finds a smaller effect than any of its studies reported. The cure is the sample size that gives 0.80 – or, failing that, reporting the estimate with its interval and refusing to headline a significant result from a design that could barely have found it.

Type S error – S for sign – is the chance that a significant result has the wrong sign: that a real rise in visits is reported as a significant fall. It is negligible at any reasonable power – under 1% in the 50-pair simulation – and becomes a live risk only when the power falls toward the α it started from, which is where a study of ten or twenty cases on a small effect sits. Both errors are why a surprising significant finding from a small study is treated as a lead to replicate, not as a result.

Reporting a plan #

Sample size determination belongs in the Method section, under its own heading, with everything a reader needs to recompute it: the test planned for, the expected effect size and its source, α and the tail, the target power, the resulting sample size, and the allowance for attrition. For the café’s branch comparison: “Sample size was planned for an independent-samples t-test with a power of 0.80 to detect d = 0.45 – the difference between branches observed in the previous season – at α = 0.05, two-tailed, giving 79 members per branch; allowing 15% for unusable records, 93 were recruited per branch.” A precision plan reports the estimate, the confidence level, the expected spread and its source, and the margin: “a 95% confidence interval for the mean spend no wider than ± 0.50, assuming an SD of 3.63 from the previous survey, requires 205 members.” “A medium effect was assumed” is not a plan, and a reviewer will say so; “no prior estimate was available, so the study was powered for the smallest effect of practical interest, d = 0.14” is. Observed power has no place in a report, and a null result is described by its confidence interval and by the effect the design was powered to detect. The planner’s reporting checklist has the list in full.

Where this appears in DataSuite #

• Analysis planner – Power analysis solves for sample size, power or the minimum detectable effect for eleven tests, with the sensitivity table, the power curve and the heatmap; the Effect size converter turns a reported measure into the one a test plans with; the Precision planning tool finds the sample size a target margin of error takes; the Group allocation optimizer prices an unequal split; Correlation attenuation corrects an expected correlation for unreliable measures, and Scale length planning says how many items a target reliability takes.
• Comparison analysis, correlation analysis and regression analysis – the effect sizes with confidence intervals that a finished study reports are the planning inputs for the next one.
• Meta-analysis – a pooled effect size is the best expected effect a plan can start from, and its interval says how uncertain the plan is.
• Reliability analysis – the coefficient that sets how much of a true correlation survives measurement, and so how much larger a sample the attenuated effect needs.
• Settings – the significance level and the multiple comparison adjustment the planned test will actually run under, which is the α to plan at.