Confidence intervals and the bootstrap #

Every number a sample gives is an estimate of a number the population has, and a different sample would have given a different estimate. A confidence interval is the honest way to write that down: not “the mean spend is 8.71” but “8.71, and anywhere from 8.20 to 9.21 is compatible with what we saw”. This page explains the estimate, the standard error that measures its wobble, the interval built from it, the meaning of the level, and the bootstrap – the resampling method the app uses to build intervals for statistics that have no formula. The numbers come from the café data, prepared as that page describes, and the intervals from descriptive statistics and comparison analysis.

The estimate and its wobble #

Parameter is the quantity the population has and no sample shows – what the café’s members as a whole spend per visit, the true gap between the branches – which the estimate stands in for. A fitted model is a set of them: each coefficient, loading, variance and covariance it contains is a parameter the fit estimates, and a model’s count of free parameters is how many numbers it has to learn from the data. As a column header, Parameter names which one the row reports – a model term, or a smoothing weight under the name R gives it (alpha, beta) – and in SEM either the variable whose variance the row holds or a line of the model syntax, whose operator says which kind of parameter it is: F =~ x1 a loading, x1 ~~ x2 a covariance, y ~ x a regression.

Estimate is the value the sample gives for a quantity the population has – a mean, a difference between two means, a correlation, a regression coefficient, a proportion. The café’s 200 members spent 8.71 per visit on average before the menu change; the 200 are the sample, and 8.71 is the estimate of what the café’s members as a whole spend. It is the best single number the data can give, and it is almost certainly not exactly right: the first twenty members in the file average 9.82, and a different twenty would average something else again. An estimate on its own says nothing about how far off it might be – that is the job of the two numbers that accompany it.

Standard error is how far an estimate typically wanders from sample to sample: the standard deviation of the estimate itself, over all the samples one could have drawn. For a mean it is the variable’s SD divided by the square root of the sample size – spend_before has an SD of 3.63, so the mean of 200 members has a standard error of 3.63 ⁄ √200 = 0.26, and the mean of 20 would have one of 0.81. That square root is the whole economics of sampling: four times the cases buy half the standard error. A standard error is not a standard deviation of the data – the SD says how much members differ from one another, and stays about the same however many are sampled; the SE says how precisely their average is known, and shrinks as more are.

The confidence interval #

Confidence interval is the range of population values compatible with the sample, at a stated level of confidence: the estimate, plus and minus a margin set by the standard error and the level. It carries what neither the estimate nor the p-value does – how large or small the quantity could plausibly be – and a report that gives the estimate and its interval has told the reader nearly everything a test could add.

95% CI is the interval at the level the app uses by default: the estimate plus or minus about two standard errors (1.97 for the café’s 200 cases, from the t distribution; 1.96 in the limit). The mean spend of 8.71 has a 95% CI of 8.20 to 9.21; the difference between the branches, 1.61, has one of 0.62 to 2.60; the correlation of 0.55 between overall satisfaction and taste, one of 0.44 to 0.64. Each says the same kind of thing: the data are compatible with a population value anywhere in this range, and not comfortably with one outside it. What “95%” means is exact and a little slippery. It is a property of the procedure: if the study were repeated many times and an interval computed each time, 95 in 100 of those intervals would contain the true value. It is not a 95% probability that the true value lies in this particular interval – the true value is fixed, and this interval either contains it or does not; what is known is that intervals built this way rarely miss. For everyday reading the distinction rarely bites: treat the interval as the range of values you cannot rule out.

Confidence level is the percentage in the interval’s name, and a trade between width and confidence. At 90% the interval for the mean spend is 8.28 to 9.13; at 95%, 8.20 to 9.21; at 99%, 8.04 to 9.37. The more confident you want to be of catching the true value, the wider the net; 95% is the convention, and the Confidence level setting changes it for every interval the app reports at once.

90%, 99% and other levels are read exactly as the 95% CI is, with the level in the header. A 99% interval is wider than a 95% one on the same data by about a third, and a 90% one narrower by a sixth.

Lower and Upper are the two ends of an interval when a table gives them in separate columns rather than as a pair.

What sets the width #

The width of an interval is the standard error times a multiplier for the level, so it shrinks with the square root of the sample size and nothing else you control. On the café data the 95% interval for the mean spend is ±0.51 around 8.71. Halving that to ±0.25 would take about 800 members, four times as many; the first twenty members alone give 9.82 ± 1.60. It is worth knowing this before a study rather than after: the analysis planner turns a target precision or a target effect into the sample size that reaches it, and the power and sample size page explains the calculation.

The interval and the p-value #

An interval and a test are two views of one calculation. When a 95% interval for a difference excludes zero, the two-tailed test of “no difference” gives p < 0.05, and when it includes zero the test does not – the branch difference’s interval of 0.62 to 2.60 sits clear of zero, and its p is 0.002; the change in visits after the menu, 0.64 with an interval of −0.003 to 1.28, just reaches zero, and its p is 0.051. The interval is the more informative of the two: it says that the visits may have risen by a little over one per member or not at all, where p = 0.051 says only “not quite”. A result that is “not significant” with an interval from −0.003 to 1.28 is a very different thing from one with an interval from −5 to 5, and only the interval can tell them apart – which is why the effect size and its interval belong beside every p in a report.

Intervals for other quantities #

Not every interval is symmetric, and not every one comes from a formula.

Interval for a median is built from the order statistics – the sorted values themselves – rather than from a standard error, and the app offers two constructions. For the café’s waiting times, median 4.9 minutes, the exact interval is 4.3 to 5.3; the descriptive statistics manual describes the two and when to prefer each.

Interval for a proportion uses the Wilson score construction, which stays inside 0 and 1 and behaves near them, where the textbook formula does not. The 135 of 200 members who would recommend the café are 67.5%, with a 95% interval of 60.7% to 73.6%.

Clopper–Pearson interval is the exact interval for a proportion: every population share from which the observed count would not have been a rare outcome, found from the binomial probabilities themselves rather than from a normal curve fitted to them. It is the interval the one-sample proportion test prints beside the observed share, because the exact binomial test is built on the same probabilities and the two read the same evidence: for the 135 of 200 members who would recommend the café it runs from 60.5% to 73.9% – a little wider than the Wilson interval’s 60.7% to 73.6% – and against a hypothesised share of 60% the test gives p = 0.030, with the interval’s lower end just clear of 60%. Its guarantee runs the other way from Wilson’s: it never covers the true share less often than its level says, at the price of covering it more often, and so being wider, at most sample sizes.

Interval for a standard deviation is Bonett’s in the app, which allows for the shape of the data: the waiting times’ SD of 2.91 has an interval of 2.47 to 3.47. The textbook interval assumes normality, and on a skewed variable like wait it would claim more confidence than it has.

Interval for an effect size is usually asymmetric. Welch’s d for the branch comparison is 0.45 with an interval of 0.17 to 0.74; a correlation’s interval is built on a transformed scale and mapped back, so 0.55’s interval of 0.44 to 0.64 reaches further down than up. An interval that is not centred on its estimate is not a mistake – it is the shape of the estimate’s uncertainty.

Wilson band is the Wilson interval for a proportion drawn at every point of an ECDF: the curve’s height at each value is the share of the sample at or below it – a proportion – so it gets a proportion’s interval, and the band those intervals trace is pointwise: each covers its own point of the true curve at the confidence level, not the curve as a whole. It is widest in the middle, where a proportion is least certain, and pinches shut at both ends, where the curve reaches 0 and 1: for the 95 Riverside waiting times the 95% band is about ±0.10 at the median and ±0.05 where the curve stands near 5%. Read it one value at a time – is the share of waits under four minutes really as high as the curve says? – and not along its length, since a band read at many points is no longer a pointwise one. It is the default Confidence band of the ECDF plot in distribution analysis and comparison analysis, named Wilson (pointwise).

DKW band is the simultaneous band for an ECDF, after Dvoretzky, Kiefer and Wolfowitz: a strip of constant half-width $\epsilon =\sqrt{\ln (2/\alpha )/(2n)}$ either side of the curve, which covers the whole true distribution function at once with at least the confidence level’s probability, as the DKW inequality guarantees. It is wider than the Wilson band nearly everywhere – ±0.14 for the 95 Riverside waits at 95%, against Wilson’s ±0.10 at its widest – and it does not narrow at the ends, because that is the price of a claim about every point at once: where a Wilson band has to be read at a single value, this one can be scanned along its length or read against another group’s whole curve. Choose it when the question is about the shape of a distribution rather than the share at one value; the plot names it DKW (simultaneous).

The bootstrap #

Some statistics have no formula for their standard error, or one that holds only for normal data. The bootstrap replaces the formula with the computer: it treats the sample as if it were the population, draws new samples from it, and watches how much the statistic moves.

Bootstrap means resampling with replacement: from the 185 waiting times, draw 185 at random, allowing the same member to be drawn twice and another not at all, and compute the statistic on that resample; then do it again, a thousand times. The medians of a thousand such resamples spread with an SD of about 0.25 – that is the bootstrap standard error of the median – and 95% of them fall between 4.3 and 5.3, which is a bootstrap 95% interval. No formula for the median’s standard error was needed and no assumption about the shape of the waits was made; the resamples carried the shape with them.

Percentile bootstrap is the simplest interval: the 2.5th and 97.5th percentiles of the resampled statistics, as above. It is what most of the app’s bootstrap intervals use, and it is marked with a superscript “b” in the table so that it is not mistaken for an interval from a formula.

BCa – bias-corrected and accelerated – adjusts the percentile interval for two things the plain version ignores: a resampling distribution that is not centred on the estimate, and one whose spread changes with the value. It is what descriptive statistics uses for the skewness and kurtosis intervals when the bootstrap method is chosen, and the waiting times show why the option exists. Their skewness is 1.27; the normal-theory interval is 0.92 to 1.62, the percentile bootstrap 0.41 to 1.89 and BCa 0.58 to 2.20. The formula’s interval is the narrowest because it assumes the variable is normally distributed – which is precisely what a skewness of 1.27 says it is not – and the bootstrap intervals are the honest ones.

Bias-corrected percentile bootstrap is the percentile interval with the first of BCa’s two corrections and not the second: recentred for a resampled distribution that does not sit evenly around the estimate, and left unadjusted for a spread that changes with the value. The share of resamples that fell below the estimate becomes a shift, z₀: with half below, z₀ is zero and the bounds are the plain 2.5th and 97.5th percentiles; when the resampled values pile up on one side – as a bounded measure’s do near its floor, where a distance correlation or a mutual information that is nearly zero can only come out higher on a resample – both percentiles slide away from the pile-up to compensate. It is the interval behind the correlation module’s Bias-corrected percentile bootstrap (any method) option, which serves every coefficient the module offers where the analytic route serves only those with a formula, and the same recentring is what the indirect effect of a mediation model carries. For overall against taste (193 members) Pearson’s r is 0.55: the formula’s interval, built on Fisher’s z scale, is 0.44 to 0.64, and a thousand resamples put the bias-corrected one at 0.43 to 0.64 – the resampled r’s sat almost evenly around the estimate, the correction had nothing to do, and the two agree to a hundredth. That is the design: on a well-behaved coefficient the option costs only time, and on the measures that have no formula it is the only interval there is.

Bootstrap replications is the number of resamples, set once in Settings for every bootstrap the app runs. The default of 100 is enough for a standard error, which is a standard deviation and settles quickly, and too few for an interval, whose endpoints are individual resamples out at the tails: at 100 replications the 2.5th percentile is the second- or third-lowest resample, and the app flags the interval to say so. A thousand is the working minimum for an interval to be quoted; the cost is time, in proportion.

Bootstrap seed decides whether two runs give the same resamples. Left empty, the app draws fresh random resamples each run and a bootstrap interval moves in its last digits from one run to the next – the 4.3 to 5.3 above came out of one run of a thousand, and yours may differ by a tenth at either end. Any integer entered in Settings fixes the sequence, so that a run can be repeated exactly and the number in a report can be reproduced.

The app reaches for the bootstrap wherever a formula is missing or untrustworthy: the standard error of the median and the skewness and kurtosis intervals in descriptive statistics, the rank-based and robust effect sizes in comparison analysis, the indirect effects of a mediation model in regression, the reliability coefficients’ intervals, factor loadings and omega in factor analysis, and the standard errors of a structural model on request. Each manual names its bootstrap paths, and the “b” marker in a table is the sign that a number was resampled rather than computed.

Reporting an interval #

Write the estimate, then the interval in brackets with its level, in the units of the estimate: “mean spend 8.71 (95% CI 8.20 to 9.21)”; “a difference of 1.61 (95% CI 0.62 to 2.60)”; “r = 0.55 (95% CI 0.44 to 0.64)”. Name the construction when there is more than one – a median interval’s method, a bootstrap interval’s replications and whether it was percentile or BCa, Bonett’s for an SD – since a reader recomputing it in another package may otherwise get a different pair of numbers. If the confidence level is not 95%, say so with every interval, not once in the methods.

Where this appears in DataSuite #

• Descriptive statistics – intervals for the mean, trimmed mean, median, proportion, SD, variance, skewness and kurtosis, with the median and shape constructions as options; standard errors beside them.
• Comparison analysis – the difference between groups with its interval on every mean-comparison test, and the effect size’s interval when Effect size confidence intervals is checked; the Clopper–Pearson interval beside the one-sample proportion test; the Wilson or DKW band on the ECDF plot.
• Distribution analysis – the ECDF plot’s Confidence band, pointwise or simultaneous.
• Correlation analysis – an interval for every coefficient.
• Regression analysis – an interval for every coefficient, and bootstrap intervals for indirect effects.
• Reliability analysis, factor analysis, structural equation modeling – bootstrap intervals for coefficients whose sampling distribution has no closed form.
• Meta-analysis – the pooled effect and each study’s, as estimate and interval, on the forest plot.
• Settings – the confidence level, bootstrap replications and bootstrap seed, which apply to every interval at once.