Resampling – method notes #

The decisions behind the bootstrap and permutation replication counts and the seeds – the sources behind the note a card prints when its count falls short, why the count that runs is always the one you set, and why an empty seed draws one. Sections follow the settings page’s.

Bootstrap replications #

At least 1,000 replications for a bootstrap interval. The note that SEM, regression and correlation analysis share quotes Efron and Tibshirani (1993, p. 188) – “at least B = 1000 replications are needed in order to sufficiently reduce the Monte Carlo sampling error” of a BCa interval – and the same book asks 500 to 1,000 of a percentile interval (p. 275), since its limits are tail quantiles, estimated from the few resamples that land in the tails; a standard error needs only about 200 (p. 52). Efron and Tibshirani (1986, p. 72) had already called 1,000 “a rough minimum” for the BC and BCa intervals. The basis is the Monte Carlo error of the limits – how far an interval from B resamples sits from the one infinitely many would give – not the interval’s coverage. Later work puts stable limits higher still: about 2,000 for a BCa interval (Efron & Narasimhan 2020), a median of 2,733 to over 6,400 replications in Andrews and Buchinsky’s (2002) simulations, 10,000 for routine use (Hesterberg 2015). So 1,000 is the floor of the published advice rather than a target, and the note names it as a recommendation, the count that ran beside it.

5,000 for reporting an indirect effect. Where a card’s intervals include an indirect effect, the note adds Preacher and Hayes’ (2008, p. 889, note 5) recommendation of “at least 5,000 resamples for final reporting, although 1,000 is probably sufficient for preliminary analyses”. Their reason is the same Monte Carlo variance – a percentile limit is read from the extremes of the resampled distribution, and a BCa interval also needs a good estimate of the acceleration – with no derivation or simulation behind the number; Hayes (2022, p. 100) keeps 5,000 as the figure his examples use. More replications do not cure the bias-corrected intervals’ known excess of Type I errors in small samples: Fritz, Taylor and MacKinnon (2012) found no effect of 1,001, 5,001 or 10,001 replications on it. Draws that are not a bootstrap – mediate()'s quasi-Bayesian simulations – carry the count alone, without either recommendation.

A count you set is never raised. The app runs the replications the setting holds, however few: no clamp, no blocked run and no silent top-up. A raised count would make the run a different one from the one requested, and a silent one would leave the card claiming a precision the user did not choose; a low count is disclosed instead, with the recommendation as interpretation. The setting’s lower limit of 10 is a guard on the input – one to nine resamples give no usable interval – and not a recommended minimum.

Permutation replications #

A permutation p cannot fall below 1/(B + 1). A p-value from B random permutations counts the observed statistic among them, $(b+1)/(B+1)$ for the b permutations at least as extreme, so that it is never zero (Phipson & Smyth 2010), and its smallest attainable value is $1/(B+1)$ – 0.001 at the default 999. When that floor lies above the significance level, no p in the table can be significant whatever the data say, so correlation analysis states the count and the floor, and with interpretation on that the p-values cannot reach the level. Under a p-value adjustment the floor has to clear α/m instead of α, the level Bonferroni holds every p to and Holm’s and the step-up methods hold the smallest. Correlation once sized its own count from the significance level – ⌈20·√k/α⌉, with the bootstrap setting as a floor – so the count that ran was neither setting’s; it now runs Permutation replications exactly, as the comparison module does, and says when that count cannot resolve the threshold. For a resampling test at a fixed level, Davidson and MacKinnon (2000) put the minimum at about 399 replications for the .05 level and 1,499 for the .01 level, below which the test loses power; the default of 999 clears the first.

Bootstrap seed #

An empty seed draws one and records it. An empty seed once handed R no seed at all, so the resamples, permutations or random starts ran off a stream nobody recorded, and the result could not be repeated – a setup card could only have said “random”, a line no reader can act on. So when the bootstrap or the reproducibility seed is empty at a run’s start, the run draws one – a uniform integer from 1 to 999,999,999 – and computes under it, and every result can be reproduced by entering the seed its card names. The draw is uniform, so its numbers are statistically equivalent to the unrecorded stream’s, though not equal to them at the same setting. Nine digits leave over a billion of headroom below R’s largest integer, 2,147,483,647, for the offsets a computation adds to give its steps separate streams. The draw is the result’s, not the setting’s: the setting stays empty, so two runs under it show no change on a setup card, and the setup card – which governs several results, each with a draw of its own – reads random, drawn per result, while each card names its own draw. Every run draws for every empty seed, used or not, so a card names a draw only when its computation used that seed.