Meta-analysis concepts #

One study is one estimate with one margin of error. A meta-analysis takes the estimates that several studies have already published and combines them into one – weighting each by how precise it is, measuring how far the studies disagree, asking whether the disagreement follows some feature of the studies, and checking whether the studies that got published look like all the studies that were run. This page explains those ideas in order: what is being pooled and how the weights work, the two models that answer two different questions, the heterogeneity statistics, moderators, studies that report more than once, small-study effects and the funnel plot, and how to read a forest plot. The café data is one study and cannot be pooled, so the numbers on this page come from a set of trials that every meta-analysis textbook uses: thirteen trials of the BCG vaccine against tuberculosis, collected by Colditz and colleagues in 1994 and shipped with the metafor package as dat.bcg. Each trial vaccinated one group and left another unvaccinated, then counted who developed tuberculosis. Copy the table below into a spreadsheet, save it as CSV and load it into the meta-analysis module: the shape detects as 2×2 event and non-event counts, and every number on this page is reproduced with the effect measure set to the log risk ratio, the other options at their defaults.

Group 1 is the vaccinated arm and group 2 the unvaccinated one; latitude is the absolute latitude of the trial site in degrees, and allocation how participants were assigned. A risk ratio below 1 means the vaccine protected.

What is being pooled #

Meta-analysis is the statistical combination of the results of several studies that asked the same question, into one estimate with one confidence interval. It does not pool the raw data – it rarely has them – but the summary each study printed: a difference in means, a correlation, a ratio of risks. Each summary is first put on one common scale as an effect size with its own margin of error, then the effect sizes are averaged with more weight on the precise ones. The thirteen BCG trials give thirteen risk ratios from 0.20 to 1.56; pooled, they say the vaccine roughly halved the risk of tuberculosis – a risk ratio of 0.49 – with a great deal of disagreement between trials that the rest of this page takes apart. In the app’s roles every study is one row of the study table, and its columns are read by role – the events and non-events of group 1 and group 2 here, a mean, SD and n per arm or an effect with its variance on another sheet; the module names the roles it found and what each row became before anything is pooled. A physician: trials → studies, vaccinated and unvaccinated → group 1 and group 2, tuberculosis cases → events; an education researcher: classroom experiments → studies, new and old curriculum → the two groups, test-score means and SDs → the arm summaries; a psychologist: published correlations between two scales → correlations with their sample sizes.

Study means one published result: one row of the table, with its own estimate and its own sample size. Usually that is one paper, but a paper that reports the same outcome in two independent samples is two studies, and a paper that reports the same sample twice – two outcomes, two time points – is one study reported twice, which matters later. Aronson 1948 followed 123 vaccinated and 139 unvaccinated children and saw 4 and 11 cases of tuberculosis; TPT Madras 1980 followed 88,391 people in each arm and saw 505 and 499. Both are one study each, and the second knows far more than the first.

Sampling error is the margin by which a study’s estimate would differ if the same study were run again on new participants – its standard error, or squared, its sampling variance. It is what a study’s sample size buys down. Aronson’s risk ratio of 0.41 carries a standard error of 0.57 on the log scale, so the true ratio could be anywhere from about 0.13 to 1.26; TPT Madras’s 1.01 carries a standard error of 0.06, and its interval runs from 0.90 to 1.14. Every study table the module reads must yield, for every row, an effect and this variance – from a reported standard error, from confidence limits, from means and SDs and group sizes, from a 2×2 table – since the variance is what the pooling weights by.

Inverse-variance weighting gives each study a weight equal to one over its sampling variance, $w_i=1/v_i$, so that a study with half the variance counts twice as much, and the pooled estimate is the weighted average

with a standard error of $1/\sqrt{\sum w_i}$ – smaller than any single study’s, because the studies’ errors partly cancel. It is the weighting that makes the pooled estimate the most precise one the studies can give, provided every study estimates the same thing. Under it, TPT Madras carries 41% of the weight among the thirteen BCG trials and Stein & Aronson 24%, while Aronson carries 0.5% and Comstock & Webster 0.3%: the two largest trials decide the answer, and the pooled risk ratio of 0.65 sits near their own 1.01 and 0.46.

Pooled estimate is the weighted average itself, with its confidence interval and a test against no effect. The module reports it on the scale the effects were pooled on – the log of a ratio, Fisher’s z of a correlation – and prints the natural-scale value beside it: a log risk ratio of −0.43 is a risk ratio of 0.65. Two pooled estimates are printed, from the two models below, and the first thing to read is whether they agree.

Two models, two questions #

Fixed-effect model assumes that every study estimates the same true effect and differs from the others only by sampling error – that if every trial had been infinitely large, all thirteen would have found the same risk ratio. Its pooled estimate is the inverse-variance average above, and its confidence interval speaks about these studies alone. On the BCG trials it gives a risk ratio of 0.65 [0.60, 0.70]: a narrow interval, dominated by the two largest trials. The name is also written common-effect model, which says the assumption better. Whether the assumption holds is what the heterogeneity statistics answer, and on the BCG trials it does not.

Random-effects model assumes that the true effects themselves differ from study to study – the vaccine really protected better in some settings than in others – and are drawn from a distribution with a mean and a spread. It estimates that spread as the between-study variance τ² and adds it to every study’s sampling variance before weighting, $w_i^{\ast }=1/(v_i+{\tau }^2)$, so the weights become more equal: a huge study can beat down its own sampling error but not the variation between studies. On the BCG trials TPT Madras drops from 41% of the weight to 10%, Aronson rises from 0.5% to 5%, and the pooled risk ratio moves from 0.65 to 0.49 [0.33, 0.73] – the small trials, which mostly found strong protection, now count. Its estimate is the mean of the distribution of true effects, and its interval is about that mean; the prediction interval is about the next study. The module estimates τ² by restricted maximum likelihood by default and, with few studies, tests the estimate on a t distribution rather than a normal one (the Knapp–Hartung adjustment): here t(12) = −3.95, p = .002. Report the random-effects estimate unless there is a reason to believe the effect is truly one – and when the two estimates differ, as 0.65 and 0.49 do, that difference is itself a finding: the large and the small trials disagree.

How much do the studies disagree? #

Heterogeneity is variation among the studies’ true effects – disagreement beyond what sampling error alone would produce. Some is always expected: no two trials recruit the same people, dose the same way or measure the same outcome identically. The question is how much, and whether it changes the conclusion. The thirteen BCG risk ratios run from 0.20 to 1.56, and the statistics below say that spread is mostly real: the trials did not all estimate one effect. Heterogeneity is not a flaw in the meta-analysis; it is often its most useful result, since the next question – why do they differ – is one no single study can ask.

Cochran’s Q is the test of heterogeneity: the weighted sum of squared distances of each study from the fixed-effect pooled estimate,

which under no heterogeneity follows a χ² distribution on k − 1 degrees of freedom, k being the number of studies. On the BCG trials Q = 152.2 on 12 degrees of freedom, p < .0001 – the trials disagree far more than sampling error explains. The test has the usual limits of a p-value: with a dozen large trials it flags disagreement that is trivially small, and with five small ones it misses disagreement that is large, so it is read together with the two quantities below, which say how much rather than whether. The card reports it as Heterogeneity test.

Between-study variance (τ²) – tau-squared – is the variance of the true effects across studies, on the scale the effects were pooled on, and its square root τ is the between-study standard deviation, readable against the effect itself. On the BCG trials τ² = 0.31 on the log risk ratio scale, so τ = 0.56: the true log risk ratios of individual trials scatter with a standard deviation of 0.56 around their mean of −0.71, which on the ratio scale puts the middle two thirds of trials between about 0.28 and 0.86. That is the size of the disagreement, in the effect’s own units; the module also prints an interval for τ² – here 0.12 to 1.11 – which is wide, as it always is with thirteen studies. The τ² estimate is what turns the fixed-effect weights into the random-effects ones, and several estimators exist – the module’s default is restricted maximum likelihood, and DerSimonian–Laird is the one most older analyses used.

I² is the share of the total variation in the observed effects that is heterogeneity rather than sampling error, $I^2=(Q-\text{df})/Q$, expressed as a percentage: 25%, 50% and 75% are the conventional marks for low, moderate and high. On the BCG trials I² = 92% [82%, 98%] – almost all of what the eye sees in the forest plot is real disagreement, not noise. It is the most quoted heterogeneity statistic and the most misread one: it is a proportion, not a size. Two meta-analyses can both show I² = 75% while one has a τ trivially small and the other a τ as large as the effect – the first is a set of very precise studies that can tell tiny differences apart, the second a set of studies whose true effects range from harm to benefit. I² also rises with the studies’ sample sizes for a fixed τ², because bigger studies have less sampling error for the heterogeneity to be compared with. Read it as “how much of the picture is real disagreement”, and read τ² or the prediction interval for how big the disagreement is.

H² is the ratio of the total variation to the sampling variation, $H^2=Q/\text{df}$: 1 means no variation beyond chance, and a value of 4 means the studies scatter twice as widely (H = 2) as their standard errors alone would make them. On the BCG trials H² = 12.9 – the trials scatter three and a half times as widely as sampling error would. It says the same thing as I² – the two are one-to-one, $I^2=(H^2-1)/H^2$ – on a scale with no ceiling.

Prediction interval for a new study is the range in which the true effect of a new study, drawn from the same population of studies, is expected to fall: the pooled estimate plus and minus a t multiple of $\sqrt{{\text{SE}}^2+{\tau }^2}$, so it carries the between-study spread as well as the uncertainty in the mean. It is wider than the confidence interval whenever τ² is above zero, and it is the honest answer to “what should I expect if I run this in my setting?”. On the BCG trials the random-effects confidence interval for the mean risk ratio is 0.33 to 0.73 – the vaccine protects on average – while the prediction interval runs from 0.13 to 1.79: a new trial might find the risk cut by seven eighths or raised by three quarters. When the prediction interval straddles the null and the confidence interval does not, as here, the average effect is real but a new study may well show none, and the paper should say so. The module draws it as a dashed whisker through the diamond on the forest plot. It is a different object from a regression prediction interval, which is about one new observation rather than one new study.

Why do they disagree? #

Study-level moderator is a feature of the studies – not of the participants within them – that might explain some of the heterogeneity: the year, the dose, the country, the population, the way the outcome was measured. In the app a moderator is any column of the study table that no role claimed; picking a categorical one makes the run a subgroup analysis, a continuous one a meta-regression, and both answer “does the effect differ with this feature?” rather than “is there an effect?”. A moderator analysis is observational even when every study was a randomised trial: nobody assigned the BCG trials to latitudes, so anything else that travels with latitude – the era, the strain, the diagnostic method – travels with it into the result. Findings about moderators are hypotheses for the next study, not causes.

Subgroup analysis splits the studies by a categorical moderator, pools each subgroup on its own, and tests whether the subgroup means differ. By how participants were assigned, the BCG trials pool to a risk ratio of 0.38 in the seven randomised trials, 0.60 in the two with alternate allocation and 0.65 in the four with systematic allocation – a pattern one might read as better-run trials finding more protection – but the test of the difference gives F(2, 10) = 0.97, p = .41, and the moderator explains none of τ². With two studies in one subgroup and four in another the analysis had almost no power, which is the usual condition of a subgroup analysis: the module withholds a subgroup with too few studies from a separate-τ² fit and counts the studies per coefficient under the moderator list so that the budget is visible before the run.

Meta-regression fits a regression line across the studies, with each study’s effect as the outcome, a continuous moderator as the predictor and the inverse-variance weights deciding how much each study pulls the line. The classic result on the BCG trials is that the vaccine worked better further from the equator: the log risk ratio falls by 0.029 for every degree of absolute latitude, t(11) = −3.55, p = .005, which is a risk ratio of about 0.88 at 13° (Madras, Frimodt-Moller) and about 0.26 at 55° (Ferguson & Simes). Latitude accounts for 76% of the between-study variance – τ² falls from 0.31 to 0.08 – and the disagreement that remains, Q = 30.7 on 11 degrees of freedom, is still more than chance. The line is drawn on the module’s Bubble plot, each study a bubble whose area is its weight. The slope is a relationship across studies: reading it as “moving a person 10° north cuts their risk” is the ecological fallacy, since no trial moved anyone. With few studies a meta-regression also overfits easily – ten or so studies per coefficient is the working guidance, which one continuous moderator on thirteen trials meets and a second would break – so moderators are pre-specified, tested sparingly and, where the studies are few, tested by permutation rather than by the t distribution.

Studies that report more than once #

Dependent effect sizes are two or more rows of the study table that come from the same participants – a trial that measured three outcomes, or the same outcome at two time points, or several subgroups of one sample. Pooling treats every row as an independent study, so such a study is counted several times, its evidence is over-weighted, and the pooled interval is narrower than the evidence supports. The BCG table has none: each trial appears once. Where a review has them, there are three answers. The naive one pools the rows as they are, and the module warns about the repeated labels. The simple one combines each study’s rows into one composite effect first, under an assumed correlation between them, so that every study counts once. The full one models the hierarchy – a random effect per study and one per row within it – and tests every estimate with a standard error robust to the clustering, at the cost of the blocks that have no form for dependent effects. The module offers all three under Dependence, shown whenever a study label repeats, and the manual’s Dependence section says which to prefer.

Are the published studies all the studies? #

Small-study effects is the pattern in which smaller studies report larger effects than bigger ones. It has several possible causes, and the tests for it cannot tell them apart: publication bias, below; genuine differences between the populations that small and large studies recruit – small trials are often run in specialist centres on sicker patients; lower methodological quality in small trials; and chance. The module tests for the pattern with Egger’s regression – the effects regressed on their standard errors – and Begg’s rank correlation, and on the BCG trials finds none: Egger’s t(11) = −0.78, p = .45, Begg’s τ = 0.03, p = .95. Both tests have almost no power below ten studies, so a non-significant result is never evidence that the literature is complete.

Publication bias is the tendency of studies with significant or striking results to be published, and of null results to stay in the drawer, so that the studies a reviewer can find are not the studies that were run. Its footprint on a meta-analysis is small-study effects – a small study with a null result is the one most likely to go unpublished, so the small studies that are found lean towards large effects – and its consequence is a pooled estimate biased away from zero. Several methods estimate what the pooled effect would be if the missing studies were present: trim-and-fill imputes the studies that would restore the funnel’s symmetry; PET–PEESE extrapolates Egger’s line to a study of infinite size; a selection model fits how much less likely a non-significant result was to be published. Each rests on a model of how studies went missing and each can be badly wrong when heterogeneity is large, so they are reported as sensitivity analyses beside the main estimate, never in its place.

Funnel plot draws each study’s effect against its standard error, with the precise studies at the top and the imprecise ones spread along the bottom, so that studies scattered by sampling error alone fill a symmetric funnel around the pooled estimate. A gap in one bottom corner – small studies present on the side of large effects and missing on the other – is the visual sign of small-study effects. The module draws the funnel where studies would fall under sampling error alone – 95% of them at the default confidence level – and shades the regions of non-significance in grey: a gap that falls inside the grey is where non-significant studies would sit, which is what publication bias leaves out, while a gap outside it points at some other cause. On the BCG plot the two big trials sit at the top, one on the null and one at 0.46, and the eleven small trials spread across the bottom on both sides of them, which is what a funnel with no asymmetry to speak of looks like; the picture is read, like the tests, as a check rather than a verdict.

Which study drives the result? #

Leave-one-out analysis refits the model once per study with that study removed and reports the range of pooled estimates the omissions produce. A pooled estimate that moves little whichever study is dropped rests on the whole set; one that changes its conclusion when one study leaves rests on that study, and the paper reports the analysis with and without it. On the BCG trials the thirteen refits give risk ratios from 0.45 (without TPT Madras) to 0.53 (without Hart & Sutherland), all well below 1: no single trial carries the conclusion. Beside it the module reports metafor’s influence measures per study and a Baujat plot – each study’s contribution to heterogeneity against its influence on the pooled estimate, so that the study in the top-right corner is the one driving both. An influential study is not a wrong one: the flag says the estimate leans on it, which is a reason to report both analyses, not a reason to drop it.

Reading the forest plot #

Forest plot shows every study as a square at its effect with a horizontal line for its confidence interval, the square’s area proportional to the study’s weight, and the pooled estimate underneath as a diamond whose width is its confidence interval. The vertical line is the null – a ratio of 1, a difference of 0 – and a study whose line crosses it did not on its own reject no effect. On the BCG plot the two big trials, TPT Madras and Stein & Aronson, draw as large squares with short lines, one on the null and one well left of it, and the small early trials as small squares with long lines mostly left of the null; the diamond sits at 0.49 and clears 1, and the dashed whisker through it – the prediction interval – reaches 1.79. A ratio is drawn on a logarithmic axis, so that a halving and a doubling sit the same distance from 1, and the axis is labelled in natural units. The plot is the first thing to look at, before any statistic: whether the studies line up, whether one big square sits apart from the rest, and whether the diamond’s story and the whisker’s story are the same.

Where this appears in DataSuite #

• Meta-analysis – the module: the study table read by role in any of eleven shapes, effect sizes converted from whatever the papers printed, both models with the heterogeneity statistics and the prediction interval, subgroup and meta-regression models, the dependence options, the funnel plot with its tests and adjustments, and the influence diagnostics.
• Effect sizes – the measures a study table is converted to: standardised mean differences, correlations, odds and risk ratios.
• Confidence intervals – the standard error every study contributes and the interval the pooled estimate carries.
• Hypothesis testing – the p-values on the pooled estimate, the heterogeneity test and the moderators, and why a permutation p is preferred with few studies.