Hypothesis testing #

Nearly every results table in DataSuite has a column headed p, and it is the most misread number in statistics. This page explains what it is, what has to be true for it to mean anything, and the handful of ideas around it: the null hypothesis, the test statistic, the significance level, tails, how to show that an effect is absent, and what happens when you run many tests at once. It assumes no background. The numbers come from the café data, prepared the way that page describes, and every one of them can be produced in comparison analysis – the equivalence section’s first instance in correlation analysis.

The question a test answers #

A test starts from a question about a population – all the people or things you care about – asked of a sample, the ones you have. The café owner cares about every member, present and future; the file holds 200 of them. A difference in the sample is a fact. Whether it says anything about the population is the question, and it is a real question because a different 200 members would have given different numbers.

In the app’s vocabulary, a two-group comparison asks whether the dependent variable – the thing measured – differs between the two levels of a grouping variable – the thing that sorts the cases into groups. On the café data: does the average spend per visit before the menu change, spend_before, differ between the members of the two branches? In the sample it does. Riverside members averaged 7.91 (standard deviation 2.97, 101 members) and Station members 9.52 (SD 4.05, 99 members) – a gap of 1.61. The question a test answers is whether a gap that size could turn up by the luck of who was sampled, from a café where the branches really do not differ.

The same question wears other nouns everywhere: a nurse comparing two wards on days to recovery (ward → grouping variable, days → dependent variable); a teacher comparing two classes on a test score (class → grouping, score → dependent); a marketer comparing two versions of a page on time spent (version → grouping, time → dependent). Learn the roles and the test does not care about the nouns.

Null hypothesis is the claim that there is nothing to find – no difference between the groups, no change between the conditions, no relationship between the variables – written H₀. Every test assumes it is true and asks how surprising the data would then be. For the café it is “Riverside and Station members spend the same per visit on average”. Tests are built on the null rather than on the claim you hope for because “nothing” is precise enough to compute from – it says exactly what to expect – while “something” is not. The logic is a proof by contradiction with a probability in it: assume nothing is going on, see how badly the data fit that, and reject the assumption if they fit badly enough.

Alternative hypothesis is what is true if the null is not – the means differ, the change is not zero, the relationship exists – written H₁. It can be two-sided, allowing a difference in either direction, or one-sided, naming the direction in advance; the choice is the tails section below.

The test statistic #

Test is the column that names the procedure that produced the row – Welch’s t-test, Shapiro–Wilk, Levene’s, Ljung–Box, log-rank – and it is read before any number on the row, because the name says which null the row’s statistic and p are about, and so which way to read them (the which null section). Several rows under one header are several checks the app ran – often on different nulls, sometimes on the same one from different angles – and each row is read against its own name. What each named test compares and when it is the right one is in the tests, by name, how the choice between them is made is the decision table, and the module’s manual has an entry for every test it can run.

Test statistic is one number that compresses the whole sample into “how far the data sit from what the null expects, in units of the noise”. For a two-group comparison it is the difference between the group means divided by how much that difference would wander from sample to sample (its standard error). Near zero, the data look like what the null predicts; far from zero, they do not. The letter the statistic wears names its family – t, F, χ², W, U, z – and the module’s manual says which test produced it.

On the café data, Welch’s t-test on spend_before by branch gives t = −3.20: the gap of 1.61 is about three of its own standard errors away from zero. The sign follows the order of the groups – the app lists Riverside first, so the statistic is Riverside minus Station, and a negative t means Station is higher.

t is the ratio of an estimate – a difference between means, a regression coefficient, a correlation – to its standard error. It is the workhorse for small and medium samples: its reference distribution is a little wider than the normal curve, by an amount set by the degrees of freedom, so it does not overstate the evidence when the noise itself was estimated from few cases.

F is a ratio of two amounts of variation: the spread between the group means to the spread within the groups (analysis of variance), or the variation a model explains to what it leaves unexplained (regression). It cannot be negative, and it grows when the groups pull apart or the model earns its keep. Its two degrees of freedom are the numerator’s and the denominator’s.

χ² (“chi-squared”) measures how far a set of observed counts is from the counts the null expects, summed over the cells of a table – whether recommending the café goes with the branch, for instance. It also compares the fit of two models, where the null is that the simpler one is adequate; in confirmatory factor analysis and SEM the χ² tests whether the model fits, and a small p there is bad news, as the which null section explains.

W is the statistic of the Wilcoxon signed-rank test, which ranks the paired differences instead of averaging them, and of the Shapiro–Wilk normality test, where it runs from 0 to 1 and a value close to 1 means the sample looks normal. Which W a table holds is in its caption.

U is the Mann–Whitney statistic: the number of times a case from one group outranks a case from the other, counted over every pair. It tests whether one group tends to have higher values than the other without assuming anything about the shape of the distributions. R and some tables call it W.

z is a statistic on the standard normal scale – the number of standard errors from the null – used when the sample is large enough that the noise is known rather than estimated: proportions, some correlation and regression coefficients, large-sample rank tests. A |z| above 1.96 is a two-tailed p below 0.05.

df, degrees of freedom, is the number of independent pieces of information behind a statistic – roughly the sample size minus the number of quantities estimated on the way – and it sets the shape of the reference distribution the p-value is read from. Two groups of 101 and 99 compared with Student’s t-test have 200 − 2 = 198 df; Welch’s t-test, which does not pool the two groups’ spreads, adjusts the figure downward when they differ, and reports 179.5 for the café branches. A df with a decimal point is not an error. For a table of counts the df are (rows − 1) × (columns − 1). Report it with the statistic, because the same t means different things at 5 df and at 500.

The p-value #

p is the probability of a test statistic at least as far from the null’s expectation as the one observed, if the null hypothesis were true. That conditional clause is the whole definition. For the café branches, p = 0.002: if the two branches really spent the same per visit, a gap of 1.61 or more – in either direction – would turn up in about two samples out of a thousand. A small p says the data are hard to reconcile with the null; it is a measure of surprise under “nothing is going on”, not of anything else.

What it is not takes longer to say, and matters more:

• It is not the probability that the null hypothesis is true. The p-value is a statement about the data given the null, not about the null given the data. p = 0.002 does not mean there is a 0.2% chance the branches are alike.
• It is not the probability that the result is “due to chance”. Every result is partly due to chance; p says how often chance alone produces something this extreme.
• It is not the size of the effect. A trivial difference reaches p < 0.001 with a large enough sample; a large one fails to reach 0.05 in a small sample. The size lives in the effect size and the confidence interval, and a report needs those beside the p.
• It is not a stable property of the finding. Another 200 members would give another p. Treat 0.002 as “strong evidence against the null in this sample”, not as a constant of nature.

Which null #

A p-value is about its null, and what you hope to see depends on which null the test holds. In a comparison the null is “no difference”, so a small p is usually what the researcher was looking for. Many tests in the app run the other way. A normality test holds the null “this variable is normally distributed”, so a small p means it is not: on the café data Shapiro–Wilk gives W = 0.93, p < 0.001 for wait, whose long right tail is plain in a histogram, and W = 0.98, p = 0.013 for spend_before, a milder departure. The same reversal holds for tests of equal variances (Levene’s, the F-test), of sphericity (Mauchly’s) and of model fit (the χ² in CFA and SEM): there, “significant” means an assumption failed or a model does not fit. Before reading a p, find out what its null is; the table’s caption and the module’s manual say.

Assumption test significance level is the threshold the app uses for these reversed tests, separately from the main significance level, because the logic runs backwards: a higher threshold (0.10 rather than 0.05) makes an assumption check stricter, since it flags more departures. It is set in Settings; the assumptions page says what each check tests and what a failure costs.

Which level a table reads follows from its null. A test whose null is that the data, an assumption or the model are fine – a normality or variance check, a fitted model’s residual diagnostics, the χ² test of exact fit and the close-fit test in CFA and SEM, the Δχ² of an invariance step, a Ljung–Box test for autocorrelation, KPSS with its null of stationarity – is judged at the assumption test significance level, so that lowering the main level for the hypotheses never makes a check more lenient; a p left unhighlighted at a lowered main level would read “fine” in the misleading direction. Inference for a hypothesis or a model – a comparison, a coefficient, a model’s F test, a unit-root test whose null is that the series is not stationary – reads the main significance level. So does a test between models you specified yourself – the Δχ² of a model comparison in SEM – since whether the added path is needed is your hypothesis, not the app’s check, where the same Δχ² in an invariance step asks whether the groups may be compared at all. The two can share a table: in the stationarity table of time series analysis KPSS reads the assumption level beside ADF and PP at the main one, and that split is what keeps the pair moving the same way when the two levels are set apart – at a main level of 0.01 and an assumption level of 0.05, an ADF p of 0.03 is no rejection (the unit root stands) and a KPSS p of 0.03 is one (stationarity falls), so both say non-stationary, where KPSS read at the main level would call the series stationary against its partner. The settings page lists the tables that read each level.

How the app shows p #

By default a p-value is printed to three decimal places, and one smaller than 0.0005 is shown as < 0.001 rather than as a row of zeros. The P-value settings can switch this to a category (p ≤ 0.05) or hide the column; significant p-values can be bold, coloured or highlighted; and the test statistic can carry stars (*, **, ***) for p at or below 0.05, 0.01 and 0.001. Where a multiple comparison adjustment is active, the formatting and the stars follow the adjusted p, not the raw one.

Significance #

Significance level is the p below which you will call a result significant – written α and fixed before the test is run. The convention is 0.05: if the null were true, a test would clear that bar by chance one time in twenty. It is a convention, not a law of nature; 0.01 buys fewer false alarms at the price of more misses, and some fields expect one or the other. The café’s branch comparison, p = 0.002, is significant at either. The app’s threshold is the Significance level setting.

Statistical significance means one thing: p came out below α. It does not mean large, important, or proven, and a “non-significant” result does not mean “no effect” – it means the data did not rule the null out; showing that an effect is absent takes an equivalence test. The word is a term of art that happens to look like an English word, and most misreadings of statistics come from taking it as one.

Interpretation is the column where the app turns a row’s numbers into words: the significance verdict against the significance level – read from the adjusted p when an adjustment is on – and, where the row carries them, the magnitude band of its effect size (small, medium, large) and the direction of the difference, so the café’s branch row reads “Significant difference, small effect, 1 < 2”. Beside a fit index or a diagnostic it is the verdict the module’s manual documents for that table. It is a reading of each number against its threshold, never a summary of the study.

Metric is the column of a key–value table that names what each row reports – a count, a ratio, a test, a threshold, a choice the run made – with the row’s result under Value beside it. Such a table is a list of different quantities rather than one quantity across cases, as a diagnostics summary is – the sample size, the share of missing values, a sampling-adequacy measure, the outliers found – so each row is read on its own terms, and each name in the column has its own entry in the module’s manual.

Value is the column of a key–value table that holds each row’s result – a number, a count, a percentage, a test written out whole (χ² with its df and p), or a word where the row reports a choice, such as the estimator that ran. Its unit and its reading are the row’s, named under Metric: two rows of one table may share nothing but the column. A coloured value is the module’s flag on that row – green for a check passed, amber for one worth a look, red for one failed – and a sentence in place of a number says why the number could not be given. The row’s entry in the module’s manual says what the value means and what sets its colour.

A result on the boundary #

Members’ visits also rose after the menu change – by 0.64 visits per member on average over the eight weeks – and the paired t-test gives t = 1.96, df = 199, p = 0.051. Not significant, by one thousandth. The wrong reading is “the number of visits did not change”. The right one is that the data leave both possibilities open: the 95% confidence interval for the change runs from −0.003 to 1.28 visits, an interval that just touches zero. A p of 0.051 and a p of 0.049 are the same evidence, and a report that says “not significant” of the first and “significant” of the second – or, worse, drops the first – is reporting the coin the sample tossed, not the café. Give the exact p, the estimate and its interval, and let the reader see how close it was. The fix for a finding that hovers at the threshold is more data, not a different test.

Two errors sit on either side of α.

Type I error is the false alarm – rejecting a null that is true – and α is its rate: at 0.05, one true null in twenty is wrongly rejected.

Type II error is the miss – keeping a null that is false – and its rate depends on how large the real effect is and how big the sample; its complement is power, the chance of finding an effect that is there – and the sample size that buys it is the power and sample size page.

One-tailed and two-tailed tests #

Two-tailed means the alternative hypothesis allows a difference in either direction, and the p-value counts both tails of the reference distribution: how often a statistic this far from zero would arise, above or below. It is the default everywhere in the app, and the honest choice unless you had a direction in mind before you saw the data.

One-tailed means the direction was fixed in advance – “Station spends more”, not “the branches differ” – and only that tail counts. The p-value in the predicted direction is half the two-tailed value; in the other direction it is close to 1, whatever the data show. On the café data the two-tailed p for the branch comparison is 0.002; with the Test direction set to One-tailed (Group 1 < Group 2) – Riverside less than Station – it is 0.0008, which the app rounds to 0.001, and with One-tailed (Group 1 > Group 2) it is 0.999. A one-tailed test is legitimate when the direction was decided before the data were seen and you would be content to report “no evidence” if the data ran the other way. Choosing the tail after looking at the means is halving the p by fiat, and a reviewer who spots it will read every other number with that in mind.

Equivalence testing #

Every test above holds the null “nothing is going on” and looks for evidence against it, and the significance section said what a failure to find any is worth: a non-significant p leaves “no effect” and “an effect too small for this sample to see” equally open. Sometimes the negative claim is the one you want – the menu change did not alter the waits, satisfaction does not depend on a member’s age – and to make it the null has to be turned around.

Equivalence test is a test whose null hypothesis is the reverse of the usual one: it holds that the effect is at least as large as a margin Δ – the groups differ by Δ or more, in one direction or the other; the correlation is Δ or stronger – so that rejecting it is positive evidence that the effect is smaller than Δ, and therefore negligible. The ordinary test and the equivalence test ask different questions of the same data – “is there evidence of an effect?” and “is there evidence that any effect is too small to matter?” – and each can come out either way whatever the other says: a difference can be significant and equivalent at once, a real effect established and shown to be smaller than Δ, or neither, from a sample too small to settle either question. In the app it is the Equivalence group of the Test direction control in comparison analysis and the Negligible-correlation margin (TOST) control in correlation analysis.

TOST, two one-sided tests, is how an equivalence test is run. The reversed null “the effect is Δ or more” is two nulls – “it is at least +Δ” and “it is at most −Δ” – and each gets a one-tailed test against its own bound; the effect is inside the band only if both are rejected, so the p-value reported is the larger of the two. It is read against the significance level like any other p: below α, the effect is shown to be smaller than Δ. Correlation analysis prints it in a column headed p (Δ) beside the ordinary p once a margin is set; comparison analysis prints it as p (equiv) with the two one-sided p-values beside it.

Equivalence margin is Δ – the largest effect you would still call negligible – and no equivalence test runs without one, because “negligible” means nothing until someone says how small is small enough. It is chosen before the data are seen and stated in the units of the effect it bounds: a difference between means in the dependent variable’s own units or in standard deviations, a correlation on the coefficient’s own −1 to 1 scale. It is the quantity the planning page calls the smallest effect size of interest, and the verdict rests on it entirely: a wide margin makes equivalence easy to show and worthless once shown, a narrow one takes a large sample to show at all, and a margin picked after the results – just wide enough to clear – invalidates the test exactly as choosing the tail after seeing the means does. Comparison analysis asks for it as the Equivalence bound (Δ), correlation analysis as the Negligible-correlation margin (TOST).

Negligible is a claim about the size of an effect – smaller than the margin – and only an equivalence test makes it. A non-significant p never does: “no significant difference” says the data did not rule the null out, which is compatible with an effect of any size the sample was too small to see. The magnitude band the Interpretation column attaches to an effect size – negligible below the small threshold – is not a test either: it reads the point estimate against a fixed bar and says nothing about how far from it the population value could lie. A TOST p below α, or the (1 − 2α) interval lying inside ±Δ, is what says “negligible” as a finding.

(1 − 2α) confidence interval is the interval whose verdict matches the TOST’s: two one-sided tests at α reject exactly when the confidence interval built at 1 − 2α – the 90% interval when α = 0.05 – lies entirely between −Δ and +Δ. It is narrower than the 95% interval the app prints elsewhere, and reading that wider interval against ±Δ understates the evidence for equivalence. Comparison analysis builds the difference’s interval at 1 − 2α in equivalence mode and names the level in the column header; correlation analysis keeps its usual interval and names the equivalence level in a caption under the table.

Non-inferiority is the claim that Group 1 is not worse than Group 2 by more than the margin Δ – the half of the equivalence test that matters when only one direction of difference would be bad news: a cheaper recipe, a shorter procedure, a new branch, each asked whether it keeps what the old one delivered. It is a single one-tailed test against one bound. With higher values the better outcome, the null is “Group 1 falls short of Group 2 by Δ or more” and the alternative is Group 1 − Group 2 > −Δ; when lower values are better – a waiting time, an error rate, a cost – the Preferred direction control flips the inequality and the alternative becomes Group 1 − Group 2 < Δ. No evidence of a difference is needed, only evidence that any difference stays on the right side of the bound, and the verdict matches the (1 − 2α) interval lying entirely above −Δ (or, flipped, below Δ). On the café’s overall satisfaction Riverside is not inferior to Station by half a point: p = 0.0006 at Δ = 0.5, the 90% interval’s lower end of −0.06 sitting well inside the bound. In the app it is the Non-inferiority entry of the Test direction control’s Equivalence group.

Superiority is the mirror claim: Group 1 beats Group 2 by at least Δ – the alternative Group 1 − Group 2 > Δ when higher values are better, Group 1 − Group 2 < −Δ when lower are – again a single one-tailed test against one bound. It differs from a plain one-tailed test in what the difference has to clear: the one-tailed test asks for any difference in the predicted direction, however small, and superiority for one worth at least the margin, so a difference that is significant and smaller than Δ passes the first and fails the second. Its interval reading is the (1 − 2α) interval lying entirely beyond Δ. On the café data Riverside’s 0.40-point lead on overall is not superiority by half a point: p = 0.643 at Δ = 0.5, the interval’s lower end of −0.06 nowhere near the 0.5 the claim needs. The entry is Superiority in the same control.

Minimal-effect test (MET) is the TOST read the other way round. Its null is that the difference lies within ±Δ – too small to matter – and its alternative that it exceeds Δ in one direction or the other: two one-sided tests again, one per bound, but now rejecting either suffices, so the p reported is the smaller of the two, and no preferred direction is asked for, since the test is symmetric in Δ. It answers “is this effect at least big enough to care about?” and is stricter than an ordinary test in the way superiority is – a significant difference smaller than Δ fails it. On the café’s overall by branch the p is 0.643 at Δ = 0.5, the same figure as superiority’s, because the upper bound is the one the 0.40 lead comes nearest to, while the lower bound’s one-sided p is 0.999. The entry is Minimal effect (MET), and its p prints in a column headed p (MET) where the other three print p (equiv).

On the café data, does a member’s age go with their overall satisfaction? Correlation analysis on the 188 members who gave both finds r = −0.06, p = 0.43: no evidence of a relationship – and, so far, none of its absence either. Set the Negligible-correlation margin (TOST) to 0.2 – a correlation weaker than that would change nothing about how the café treats anyone – and the column p (Δ) reads 0.024: the relationship is shown to be negligible at α = 0.05. Set the margin to 0.1 instead and p (Δ) is 0.28, so the claim cannot be made. The interval says the same: the 90% interval runs from −0.18 to 0.06, inside ±0.2 but not inside ±0.1, because a sample of 188 cannot tell a correlation of −0.06 from one of −0.15. Both readings are correct; which one the report carries depends on the margin, and the margin was the owner’s decision, made before the data, not the data’s.

The same reversal in comparison analysis: does overall satisfaction differ between the branches? Welch’s t-test on the 193 members who gave a score finds Riverside 0.40 points higher – 7.49 against 7.10 – with t(169.8) = 1.45, p = 0.148: no evidence of a difference, and none yet of its absence. Set the Test direction to Equivalence (TOST) with a Raw bound of 0.5 – half a point on the 0–10 scale being the most the owner would let the branches drift apart – and p (equiv) reads 0.357: not shown equivalent. At Δ = 1 it reads 0.015, and the branches are shown to differ by less than a point. The 90% interval for the difference, −0.06 to 0.85, says the same – inside ±1, not inside ±0.5 – and, as with the correlation, the margin and not the data decides which claim the report carries.

Many tests at once #

At α = 0.05 each true null has a one-in-twenty chance of a false alarm. Run six tests on six true nulls and the chance that at least one of them comes out “significant” is 1 − 0.95⁶ = 26%; run twenty and it is 64%. A study that tests many things and reports the ones that reached 0.05 is, without any dishonesty, reporting mostly noise.

The café survey has six items, and comparing all six between the branches with Welch’s t-test is one run of comparison analysis with six dependent variables – and six tests:

Read raw, two of the six are significant: Station’s members rate the staff less friendly and the service less quick. The three food items and the reverse-keyed slow show nothing. But six tests carry a one-in-four chance of at least one false alarm when nothing is going on, so the raw column cannot be read as if each test stood alone.

Multiple comparison adjustment raises the bar for each test so that the family of tests as a whole keeps the error rate you chose. The app applies the method selected in Settings to every family it recognises – the dependent variables of one run, the pairs of a post-hoc table, the cells of a correlation matrix – and can show the adjusted p beside the raw one or in its place. The methods differ in what they promise:

Family-wise error rate is the probability of at least one false alarm anywhere in the family. Controlling it at 0.05 means that in a study where nothing is going on, the chance of reporting even one significant result is 5%. Bonferroni, Holm, Hochberg and Hommel control it.

False discovery rate is the expected share of the significant results that are false alarms. Controlling it at 0.05 means that among the results you call significant, about one in twenty is expected to be noise – a weaker guarantee that keeps far more power when many effects are real. Benjamini-Hochberg and Benjamini-Yekutieli control it.

• Bonferroni – multiply every p by the number of tests (and cap at 1). The oldest and the most conservative; with many tests it hides real effects. On the survey items, 0.095 and 0.076.
• Holm – sort the p-values from smallest to largest and multiply the smallest by the number of tests, the next by one fewer, and so on, stopping at the first that fails. It gives exactly Bonferroni’s guarantee and is never less powerful, so there is no reason to prefer Bonferroni to it except a reviewer’s habit. On the survey items, 0.076 and 0.079.
• Hochberg – Holm run from the largest p downward, slightly more powerful, valid when the tests are independent or positively related – which comparisons of the same groups on related outcomes usually are.
• Hommel – a further refinement of Hochberg under the same condition, a little more powerful again and harder to explain.
• Benjamini-Hochberg (FDR) – controls the false discovery rate: sort the p-values, and the k-th smallest is significant if it is below k/m × α. The standard choice for exploratory work with many tests, where missing real effects costs more than a few false leads. On the survey items, 0.047 and 0.047.
• Benjamini-Yekutieli (FDR) – Benjamini-Hochberg’s guarantee under any pattern of dependence between the tests, paid for with a larger correction. For families whose tests may be negatively related.

p (adj) is the adjusted p-value: the raw p after the chosen method, to be compared with the significance level exactly as the raw one would be. When it is shown, the app’s Significant verdicts and its significance stars follow it, not the raw column.

Which adjustment to use #

Holm is the default worth having: it does what Bonferroni does with more power, and it needs no assumption about how the tests relate. Benjamini-Hochberg is the choice for exploratory work – a screen of many outcomes, a large correlation matrix – where a few false leads among many discoveries are the acceptable price of not missing the real ones. Bonferroni is what some fields and reviewers expect, and there is no harm in it for a family of two or three. What matters more than the pick is that it is made before the results are seen and stated in the report, with the size of the family it was applied to.

On the survey items the choice decides the finding. Under Holm neither service item clears 0.05 (0.076, 0.079); under Benjamini-Hochberg both do (0.047). That is not a contradiction: the two methods promise different things, and the data sit in the band where the promises differ. The honest report gives both the raw and the adjusted values and says which guarantee was chosen: “Station’s members rated the staff as less friendly and the service as less quick (p = 0.016 and 0.013 unadjusted); with Holm adjustment across the six items, neither reached significance (p = 0.079, 0.076).” A finding that survives only under the most lenient adjustment is a lead to follow up with new data, not a result.

Reporting a test #

A test is reported with its statistic and degrees of freedom, its p-value exactly (to two or three decimals, or < 0.001), the direction and size of the effect with a confidence interval, the sample size, and whatever choices shaped the p: the tail, the significance level if it was not 0.05, and the adjustment with the size of its family. The café’s two findings, in that form:

• Station members spent more per visit than Riverside members before the menu change – 9.52 versus 7.91, a difference of 1.61 (95% CI 0.62 to 2.60); Welch’s t(179.5) = −3.20, p = 0.002, n = 99 and 101.
• Spend per visit rose after the change by 1.09 on average (SD of the change 2.86, 95% CI 0.69 to 1.49); paired t(198) = 5.37, p < 0.001, n = 199 – one member whose after-window average was a catering order was excluded, as the café data page describes.

The comparison analysis manual has the full checklist for its tests, and every module’s manual has one for its own.

Where this appears in DataSuite #

• Comparison analysis – every test table carries the statistic, df, p and the interpretation; the Test direction control sets the tail, and its Equivalence group runs the TOST, non-inferiority, superiority and the minimal-effect test against the Equivalence bound (Δ); several dependent variables or post-hoc pairs form a family the adjustment applies to.
• Correlation analysis – each coefficient’s p tests the null “no relationship”; the Negligible-correlation margin (TOST) turns that null around for every pair and prints the result as p (Δ); a matrix of coefficients is a family, and the module warns when no adjustment is set.
• Regression analysis – each coefficient’s t (or z) and p test “this predictor adds nothing given the others”; the model’s F tests “no predictor adds anything”.
• Distribution analysis – the normality tests, where the null is “normal” and a small p is a departure.
• Factor analysis and structural equation modeling and confirmatory factor analysis – the χ² of model fit, whose null is “the model fits”, and which is significant for almost any model on a large enough sample.
• Time series, time-to-event and meta-analysis – stationarity, log-rank and heterogeneity tests, all read as above once their null is known.
• Settings – the significance level, the assumption test significance level, the p-value display and the multiple comparison adjustment, which apply everywhere at once.