Meta-analysis – method notes #

The decisions behind the meta-analysis module – which rule, threshold, refusal or fallback stands behind each card, and the reasoning that put it there. Sections follow the manual’s.

The study table #

A sheet is read through a named shape, matched by column names over its whole role set, and the shapes are tried from the most specific down. Extraction sheets spell the same quantity a dozen ways, and the arithmetic that turns a row into an effect size depends entirely on what the row holds – a mean and SD per arm is one formula, a 2×2 table another, a reported effect with its limits a third. Naming the shape up front makes that conversion a visible thing: the roles show which columns feed it, a role the matcher got wrong can be moved, and the Study rows panel says what each row became before anything is pooled. A shape holds only when every one of its roles lands on a distinct column – a half-matched shape is not the shape – and the list is walked in a fixed order, the first shape that holds winning. The precomputed shapes sit above the arm-level ones on purpose: a table already carrying an effect and its variance is never recomputed from arms that happen to sit beside them. The mixed shape sits last and is the only one that resolves partially – it holds as soon as one conversion’s roles land – so it can win only where no fixed shape read the table at all.

Numeric type is checked after the match, never as part of it. A shape that matches by name but lands a role on a column that does not parse as numbers is reported as exactly that, with the columns named, rather than passed over for the next shape: falling through is what silently pools nine studies out of fourteen, and the user’s spreadsheet parsing the way they meant it to is the one thing this module has no other feedback on. The same verdict gates the run, so a problem the detection line names is one a run refuses in the same words.

A role several columns fit takes the column spelled as its own name, then the first in selection order; the study label is matched in tiers, id last. Within the columns a role’s pattern fits, the one spelled exactly as the role’s key – metafor’s argument name, which a metafor-shaped sheet uses verbatim – outranks the order the columns were selected in, which is otherwise all that separates a d column from the yi two columns over. The others are listed on the detection line as also fits, since a claim standing on preference alone is one the reader should see the alternatives to. The study label reads through three tiers – a name that is a study’s (study, trial, author), one that probably is (label, name, reference), and id as a last resort – so that an exported table’s row id, first on the sheet, does not take the label from the trial column beside it.

A hand-picked column displaces the role that matched it by name, and two hand-picked roles on one column stop the run. Detection fills around a pin rather than over it, which is what lets a variable-selection change re-run without discarding the assignment the user made. A column the user assigns outranks name-matching’s claim on it, so the role that merely matched it gives it up; two pins on one column are the user’s own contradiction and are refused before the run, because two roles on one column reach escalc as the same argument twice – an arm compared with itself – which pools a confident nothing rather than failing.

Only a readable flip reverses a row’s direction; anything else keeps the study’s own sign. A negative number, or a cell reading -, flip, flipped, reverse, reversed, invert, inverted, negative or neg in any case, negates the row’s effect after conversion. A blank, a positive number or a word not on that list leaves the effect as the study reported it, because a cell nobody can read must not silently negate a study; the card counts the rows it reversed so the decision is visible.

A clustered row’s variance is inflated by the design effect 1 + (M − 1) × ICC, and a cluster size below 1 is not read. A cluster-randomised study analysed as though individuals were randomised reports a variance too small by that factor (Donner & Klar 2002), M being the average cluster size and ICC the intracluster correlation – the row’s own where the sheet carries one, the assumed value otherwise. The correction is applied to the row after its conversion, beside the direction sign, because it is a statement about the study rather than about the recipe that read it. A cluster of one is a study that randomised individuals, and a design effect below one would shrink a variance instead of inflating it, so a cluster-size cell below 1 leaves the row uncorrected.

When a fixed shape wins, the detection line says how many rows the mixed shape would read instead. Shapes are matched on column names alone and the most specific wins, so a sheet carrying effect, ci_lower and ci_upper columns resolves as precomputed limits however few of its rows fill them; a mixed extraction sheet has those columns for the studies that reported a ratio and leaves them blank for the rest. The count of rows the mixed shape would read is the only thing on the card that tells the user their other rows are readable, so it is printed whenever it exceeds the winning shape’s usable count.

A row is checked before conversion, and each refusal names the arithmetic it would break. Every conversion has values it cannot take, and metafor’s response to one is to fail the whole frame or to return a nonsense effect, so the panel refuses the row first and says why. A count below zero; a correlation outside −1 to 1, or of exactly ±1, which has no finite Fisher’s z or d; a p value of exactly zero or beyond ±1 (a p’s sign is read as its effect’s direction, so ±1 is a d of exactly zero, and zero has no t); a p beside its own reported estimate that is negative or exactly 1 – the estimate carries the sign, conv.wald fails the whole frame on a signed p, and z = 0 at p = 1 leaves the variance infinite – or an estimate of exactly zero beside a p, which no p can put a variance on; more events than the group holds; a group whose every cell is zero; confidence limits out of order, or an estimate outside its own limits; a zero SD in both arms, or in the one baseline SD a change standardised by the baseline divides by; a reported standard error or variance of zero, which leaves no variance to weight by; fewer than four participants in total under a standardised mean difference, where Hedges’ correction J(m) is undefined at m = n₁ + n₂ − 2 ≤ 1, fewer than three under a standardised mean change for the same reason, and fewer than four under a Fisher’s z, whose variance is 1/(n − 3); a mean that is not positive under a ratio of means, a ratio on its own scale that is not positive, and a proportion outside 0 to 1 where the values were entered on the natural scale.

A five-number summary goes through conv.fivenum’s skewness test, and an arm with n below 5 or its numbers out of order takes no five-number conversion. The estimators of a mean and SD from a median with quartiles or a range assume a distribution near enough to normal, so every five-number arm is run through metafor’s conv.fivenum with test = TRUE – the skewness test of Shi and colleagues – and an arm the test refuses is reported back by row and arm rather than converted on an assumption the data contradicts. conv.fivenum itself excludes an arm with n below 5 before estimating anything and stops the whole frame on five numbers that are not non-decreasing, so the panel applies both checks up front and resolves such an arm through no five-number kind.

A row read short of a fuller recipe, or with an estimate off the centre of its limits, resolves and is still held for attention. A baseline with one cell blank is a typo or a gap as often as a paper that reported one arm’s baseline, so a row that fills only part of the columns a fuller conversion needs is read by the one it does fill and named under the filter with what it lacks. An estimate further from the midpoint of its confidence limits than a tenth of their width – metafor’s cifac, the point at which conv.wald warns – is not the centre of a symmetric interval: it is an exact or a bootstrap interval, or a ratio entered on its own scale, as often as a rounding artefact. The row still converts, as a symmetric interval, and is flagged so that the reader can check which it was.

Effect measures #

Ratios are pooled on the log scale, correlations on Fisher’s z and proportions as log odds, and the card prints the natural-scale value beside each. An odds ratio of 0.5 and one of 2 are the same size in opposite directions, but their arithmetic distance from 1 is not; on the log scale they are symmetric (−0.69 and +0.69), the sampling distribution is close to normal, and an interval built there back-transforms to the asymmetric interval a ratio should carry. Fisher’s z does the same for a correlation, whose sampling distribution skews as r approaches ±1 and whose variance on the z scale, 1/(n − 3), no longer depends on the correlation itself; the logit does it for a proportion, whose variance p(1 − p)/n shrinks to nothing at 0 and 1, which is why the raw proportion is offered but the logit is the safer choice near the boundaries. The forest is drawn on a log axis in natural units for the multiplicative measures; a logit and a Fisher’s z keep it on the analysis scale, the only one their intervals are symmetric on.

Every row the module converts itself is Hedges’ g; a reported d is corrected on entry and a reported g is not. escalc’s SMD applies the small-sample correction J(m) by default to every row it converts from means, a t, a p or a point-biserial r, so all of them sit on one scale; a d typed into the mixed shape’s Cohen’s d column is corrected the same way, and the Hedges’ g column is passed through with correct = FALSE. The two columns exist because the correction is not idempotent: a g entered as d is corrected twice, a d entered as g goes uncorrected, and neither error is detectable from the numbers. A reported effect with its standard error, limits or p is pooled as reported under SMD, with no correction, since the module cannot know which it was given – the card says so under what it took on trust.

Each route from a printed number to an effect size is a named recipe with its own assumptions, tried in a fixed order, and the card prints the ones that fired. Each route rests on assumptions – that a t came from a pooled-variance independent test, that an interval is Wald and symmetric at the assumed level, that a p is two-sided and exact – and a single formula would hide which held for which row. The registry makes each route a named thing whose assumptions are sentences, so the result card prints exactly the ones that fired on the sheet and the write-up can say what was taken on trust. The order encodes preference where a row fills several: the pre-post-control reading before the two arm summaries, since its cells are a superset and a row that fills its baselines was extracted for the change; the arm summaries before the test statistics, since the summaries are the data the t was computed from; a standard error before an interval before a p, each lossier than the last. A row several recipes fit is read by the first and offered the rest in the Study rows panel, and the pick is remembered by case number so a filter or a re-detection does not discard it. There is no F recipe: metafor 5.0-1 declares an fi argument on escalc but its SMD branch reads di, ti and pi only, so an F from a two-group comparison enters as t = √F.

An arm’s mean and SD are reached by the reported pair first, then derived from a standard error, an interval or a five-number kind, and the derivations are the Cochrane Handbook’s. SD = SE × √n, and SD = √n × (upper − lower) / (2t) with t on n − 1 degrees of freedom at the published level (Handbook §6.5.2.2–3), are arithmetic; the interval is read as t-based because a paper’s interval on a mean is one, and a z or a bootstrap interval reconstructs a different SD, which the card’s trust block says. The five-number kinds go through metafor’s conv.fivenum with dist = "norm" and its skewness test, as the study-table notes say. A conversion reading a strict subset of another’s cells is the fallback for an arm that lacks the rest, not a competing reading of it – a five-number summary is not ambiguous between itself, its quartiles and its range – while a mean beside both an SD and an SE is a tie and gets a picker.

A precomputed interval is read at the level the user names, and the natural-scale reading is offered on the limits shape alone. A published interval carries no record of the level it was computed at, so the module asks – 95% is the default – and derives the variance as the square of the half-width divided by the normal quantile at that level; a level wrong by one step scales every standard error by the ratio of the quantiles (1.96 against 1.645 or 2.576), which is why the card reports the level back. A ratio, a correlation or a proportion quoted as itself with its limits can be carried to the analysis scale before the variance is derived, as conv.wald’s transf does for the mixed shape’s ratio and correlation recipes; a variance or a standard error quoted on the natural scale cannot, without the delta method on an estimate the module would have to trust, so those two shapes read their values on the analysis scale alone and say so. A column headed or, rr, hr or irr says what its values are, so the measure and the natural-scale reading follow it once per header seen, and a pick made after that stands.

Options #

The study table is read under pairwise missing-data handling, and the module warns under either other method. A blank cell here means the paper reported no such statistic, not that a case lost a value, so imputation fills the blanks from the column – inventing statistics no study reported – and listwise deletion drops every row carrying one, which a study table is expected to have. Neither is visible anywhere else: the setting is global and applied before this module reads anything, and every value it reads is post-imputation and post-deletion, so the notices under Study data are the only warning the sheet gets.

A double-zero table is left out under a ratio by default and kept under a risk difference; under Peto it cannot be kept. A table with no events, or only events, in both arms carries no information about a ratio, and keeping it under the continuity correction pools a ratio of exactly 1 with a large variance – a pull toward the null decided by the correction rather than by the data; the Cochrane Handbook (Higgins et al. 2019, §10.4.4) and metafor’s own rma.mh and rma.peto drop such a table, and only escalc keeps it. Two risks of zero are a statement about their difference, so under RD the default keeps the table. Peto’s one-step estimate is computed on the uncorrected cells (to = "none"), and such a table then has V = 0 and converts to NA whatever the choice, so the select is pinned there. The default is a statement about the measure, so the select resets to it whenever the measure changes. A zero in a single cell is a different matter: escalc’s default puts 0.5 on every cell of that table (add = 1/2, to = "only0") under every 2×2 measure but Peto, and the card counts the tables it corrected.

A study’s rows fold into one composite through metafor’s aggregate at the assumed within-study correlation, and the hierarchy is the CHE model under CR2 cluster-robust inference. The composite is the inverse-variance weighted combination of a study’s rows under a compound-symmetric covariance at ρ, so each study counts once and every downstream block runs unchanged on the composites; a moderator that varies within a study has no value on the composite, so it is withheld rather than letting the fold pick one by row order. The hierarchy is the correlated-hierarchical-effects model of Pustejovsky and Tipton (2022): vcalc fills each study’s block of V at ρ, rma.mv puts a random effect on the study and on the row within it – REML being the only estimator it offers the variance components – and robust with clubSandwich’s CR2 sandwich over studies tests every estimate with a t on Satterthwaite degrees of freedom (Tipton 2015), each model carrying its own fractional df; below four such df the card says the t is not to be relied on. The common-effect working model on the same V under the same robust inference stands where the fixed-effect fit stands (Hedges, Tipton and Johnson 2010). The robust inference is what stands in for Knapp–Hartung and the permutation test – permutest has no rma.mv method, and the adjustment corrects a model that took the rows as independent – and Begg’s test, trim-and-fill, the selection model, the row-level influence measures and the Baujat plot have no form for dependent effects, so they are off; Egger’s regression, PET-PEESE and the leave-one-out run on the hierarchy. The composite is the recommendation with a handful of duplicated studies and one main outcome, the hierarchy where many studies contribute many rows, since it uses all of the information; pooling every row as a study is offered so the naive analysis can be seen, not reported.

The assumed defaults are an ICC of 0.05, a pre–post correlation of 0.5 and a within-study correlation of 0.8, each on a domain the arithmetic can take, and every one in play is swept. The ICC and the pre–post correlation are conventional fallbacks for numbers papers rarely print, with no single source; the within-study correlation is the robust-variance literature’s convention (Tanner-Smith and Tipton 2014). The pre–post correlation runs from −1 to just below 1, since at exactly 1 the change score’s SD is zero and escalc recodes the row to NA; the within-study correlation from 0 to just below 1, because a negative ρ would make a study’s rows more precise combined than independent, past −1/(k − 1) the study’s block of V is not positive definite, and at exactly 1 it is singular. A typed value or a row’s own cell outside the domain is not a value at all and falls back the way a blank does, counted apart in the tally. Because none of these numbers was reported, every one in play is re-run over a grid – 0.01, 0.05, 0.1 and 0.2 for the ICC, 0.2, 0.5 and 0.8 for the two correlations, plus the run’s own default so that one point equals the main result – varying only the default: a row that carried its own value keeps it at every point, so the sweep tests what was assumed and never what was known.

REML is the default τ² estimator and Knapp–Hartung is on by default; the fixed-effect fit stays on z, and the prediction interval takes t on k − 2. REML is the recommendation of the estimator comparisons (Veroniki et al. 2016; Langan et al. 2019), approximately unbiased and stable across the range of τ²; DerSimonian–Laird is offered because most published meta-analyses used it and reproducing one is a legitimate need, though it underestimates τ² when heterogeneity is large; Paule–Mandel and Sidik–Jonkman are the two other estimators metafor’s rma runs. None of the three has a likelihood, so under them the moderator model’s likelihood-ratio comparison – REML refitted on ML for the test – is not run, since metafor answers anova() on such a fit with a warning and an invalid test rather than an error. The Knapp–Hartung adjustment (test = "knha") tests the random-effects estimate and each moderator coefficient on a t distribution with an adjusted standard error, which holds the error rate near nominal with few studies where the z test does not; it is not applied to the fixed-effect fit, which metafor warns it was not meant for. The prediction interval is on t with k − 2 degrees of freedom (Higgins, Thompson and Spiegelhalter 2009; Riley, Higgins and Deeks 2011) under Knapp–Hartung as well, where metafor’s default would take k − 1.

Moderators #

A moderator model wants ten or so studies per coefficient, a categorical moderator costs L − 1, and the budget line makes the cost visible as the picks are made. The ten-per-coefficient rule is borrowed from simple regression, where the denominator is predictor parameters, so the intercept is not among them and a factor with L levels costs its L − 1 contrasts; ticking a four-level moderator moves the count by three, which teaches the rule at the moment it costs something rather than in prose beside the list. The budget is computed on the rows a run would pool that also carry the moderator, and under the composite or hierarchy modes on the studies those rows fold to, since a coefficient is spent per study there. Under the screen with several moderators the binding constraint is the tightest single model rather than the one they would share, and the line names it. A categorical taking one level on the pooled rows adds no coefficient and is not costed – dividing by its zero would report an unbounded ratio as comfort.

The reference level defaults to the largest subgroup, and a continuous moderator is centred on its mean by default. The largest subgroup is the most stable base to contrast against, and choosing it surfaces the thin levels as a side effect; a picked level absent from the rows the model is actually fitted on would fail relevel ahead of the error guard, so it falls back to the default too. An uncentred intercept is the effect at a covariate of zero, which is rarely a quantity anyone asked for; centring puts it at the average study, and the bubble plot’s grid is shifted back onto the moderator’s own scale. Neither reaches the screen, which prints no coefficients: both move an intercept nothing there prints.

A subgroup under three studies reports no τ² of its own and stays out of the between-subgroups Q, a screen row needs three studies, and a moderator constant on the modelled rows is left out and named. rma on one study returns a τ² of exactly zero it never estimated, and on two a variance no reader should weigh, so a level under three studies – its clusters under the hierarchy – is fitted for its estimate but its τ² and I² are withheld and the level stays out of the Q over the subgroup estimates, which is Cochran’s Q on the separate fits, a fixed-effect rma over their estimates and standard errors: a different test from the omnibus of the pooled-τ² model, which a per-subgroup τ² no longer answers to. A one-moderator screen model spends two coefficients, so below three studies a row has no residual to test against, and a moderator present on no pooled study would otherwise reach R as an empty column and take the whole screen down rather than its own row. A moderator constant on the modelled rows has nothing to explain, and R would not say so – a one-level factor fails the fit, while a constant covariate is dropped by metafor into an intercept-only model whose omnibus is a test of the mean, reported as if it had been fitted – so the test is made on what reaches R, the moderator is named as constant and the model is fitted on the rest.

The permutation test runs every distinct permutation when the setting asks for at least as many, and its p floors at one over the reshuffles that ran. metafor’s permutest refers the fit to the moderator’s own reshuffled values rather than to a theoretical distribution, which is the inference that still holds at the k a meta-analysis usually has; with no more than iter distinct permutations it runs every one, so the count the card reports is the number that ran rather than the number asked for, and the smallest p it can report is one over that count. Under the screen the switch fires per moderator, since each row has its own k. The count is the global permutation replications setting, and with a seed set the reshuffle is seeded from it, so a rerun reproduces the p.

Reading results #

The two models are tabled together under one heading, and the intervals on τ², I² and H² are Q-profile intervals off one confint call. Two pooled estimates side by side with nothing between them are read as one estimate computed two ways – the most common way this analysis is misread – so the card names the model in the first column of one table and links the two readings rather than printing two cards. The interval on τ² is the Q-profile interval (Viechtbauer 2007), whose coverage holds where a Wald interval on a variance does not, and the I² and H² bounds are the τ² bounds carried through the identities that define them, so the three intervals make one statement; each can fail on a fit that itself converged, so a failed interval leaves its cell without one rather than taking the run down. Under the hierarchy the variance components are printed without intervals, and the two I² are Cheung’s (2014) multilevel I²: each component against the sum of both plus the typical sampling variance of Higgins and Thompson (2002).

Egger’s and Begg’s tests run from three studies with a caution below ten; the sample-size form of Egger’s runs where every row is a Hedges’ g with both group sizes, Peters’ where every row came off a 2×2 table, and the standard form stays beside them. A residual degree of freedom is the floor the regression needs at all, and the standing guidance is not to read either test below ten studies (Sterne et al. 2011), so the block runs from three and says at fewer than ten that a non-significant result is not evidence of symmetry. Egger’s regression regresses the effects on their standard errors, and on two measures the standard error is itself a function of the estimate – Hedges’ g’s variance carries g², a log odds ratio’s the event counts – so the standard form flags asymmetry too often under the null. Where every row is an SMD with both group sizes the same mixed-effects regression is run a second time on $\sqrt{1/n_1+1/n_2}$, the standard error the measure would have at an effect of zero, a function of the sizes alone (Pustejovsky and Rodgers 2019); where every row carries events and a total, Peters’ test regresses on 1/n weighted by the events and non-events (Peters et al. 2006). The standard form stays in the table so that a number another package prints is still here, and the note says which row to read. A funnel alone invites the reader to see asymmetry, and the two tests disagree often enough that showing one is a choice, so Egger’s and Begg’s always run together. The funnel’s significance contours (Peters et al. 2008) sit around zero, so they are drawn only where the measure has its null there – not for a proportion or a raw mean.

PET is Egger’s intercept, tested one-sided on the side of the pooled estimate at .05, and PEESE replaces it where that test rejects zero. The intercept of the regression on the standard error is the effect predicted at a standard error of zero – the regression limit estimate – and Stanley and Doucouliagos (2014) make it the corrected estimate (PET) unless it rejects zero, in which case PET overcorrects an effect that is there and the intercept of the same regression on the variance (PEESE) replaces it. The test is one-sided on the side of the pooled estimate because the question is whether an effect in the observed direction survives, and the .05 is the rule’s own. The mixed-effects fits – Egger’s, its sample-size form, PET, PEESE and trim-and-fill’s refit – carry the Knapp–Hartung test and the level of the pooled model, so each statistic off them is a t on k − 2 degrees of freedom; the selection model has no Knapp–Hartung form and stays on z, and Peters’ is the t of an ordinary weighted least-squares fit.

Trim-and-fill runs metafor’s L0 estimator on the random-effects fit and hands back the studies it imputed; the selection model is a one-step function at p = .025, one-sided on the side of the pooled estimate. trimfill estimates how many studies are missing from the thinner side of the funnel with the L0 estimator (Duval and Tweedie 2000), its default, on the side it judges from the regression test, and refits the filled set at the pooled model’s level and test; the imputed studies come back with the adjusted estimate so that the funnel can draw what was invented rather than only what it changed, and the note names the τ² estimator the refit used. The selection model is Vevea and Hedges’ (1995) step function with one step at a one-sided p of .025 – the two-sided .05 that publication decisions are made at – estimating δ, the relative probability that a study past the step was published; the side is the pooled estimate’s, since selmodel’s default assumes a positive effect and empties the significant interval on a protective one. An interval with no study in it leaves δ unidentified: selmodel still returns, with δ at its ceiling and a warning, so the card reads the warning as the model not holding and prints the counts without the fit.

metafor’s influence rule flags a study on DFFITS beyond 3√(p ⁄ (k − p)), Cook’s distance above the median of χ² on p degrees of freedom, leverage above 3p ⁄ k or a DFBETAS beyond 1; the block runs from three studies, and the Baujat coordinates are computed rather than read off baujat(). The cutoffs are influence.rma.uni’s own (Viechtbauer and Cheung 2010), computed on the fit’s k and p and printed in the note so that a flag can be checked against them; the card picks none of its own. A leave-one-out fit needs a pool left behind – at two studies the refit is a single study, with no heterogeneity to report and no interval worth reading – so the block runs from three. Both batteries refit the model once per study, which is why the block is the slowest with many studies. Baujat’s two coordinates – each study’s squared standardised residual against the squared shift the pooled estimate takes without it, over the refit’s squared standard error – are computed from the leave-one-out refits rather than read off baujat(), which yields them only as a side effect of drawing, and a draw would open a graphics device on the engine the whole app shares; they are the same quantities to machine precision. Either battery can fail on a fit that itself converged, so each stands alone and the note names the one that did not run.

Under the hierarchy each study is left out with all of its rows at once, and its Cook’s distance comes off the same refit, started from the full fit’s variance components. The unit that can drive a dependent-effects result is the study, and leaving out one of a study’s rows moves nothing that leaving out the study does not, so the row-level measures, the row-by-row leave-one-out and the Baujat plot are off. Each refit is rma.mv on the remaining studies, started from the full fit’s σ² as cooks.distance(cluster = study) starts its own, so the leave-one-out row and the study’s Cook’s distance – the squared shift of the estimate over the full fit’s variance of it – come from one refit per study rather than two, and the distance is that call’s number to machine precision; the interval on each refit is the cluster-robust one the pooled estimate carries, and the cutoff is the χ² median on p degrees of freedom, as on the independent path.