Test assumptions #

Every test in the app has conditions under which its p-value and its confidence interval mean what they say, and the Check assumptions button in comparison analysis and correlation analysis, and the diagnostics of regression analysis, exist to tell you whether those conditions hold on your data. This page is about the conditions themselves: what an assumption is and what breaking one costs, then each of the assumptions the app checks – independence, normality in brief, equal variances, the straight-line assumptions of a regression, sphericity, equal covariance matrices, the two assumptions a covariate adds, and the small-sample conditions of the categorical tests – and finally how to act on a failure. The numbers come from the café data, prepared as that page describes; where the app runs the same check, its card prints the same values.

What an assumption is #

Assumption is a condition a test takes for granted about how the data came about, in exchange for which its arithmetic becomes possible: the p-value and the interval are computed for data that arose that way, and when it did not they describe a situation that did not occur. Every test has some – the parametric tests more, the rank tests fewer, none of them zero – and they come in three kinds. Assumptions about the design, chiefly independence, are settled by how the data was collected and cannot be tested from the numbers. Assumptions about the population – normality, equal variances – concern the shape of what the values came from and can be checked on the sample, roughly. Assumptions about the model – linearity, sphericity, parallel slopes – concern the form the analysis imposes. In the app’s role vocabulary, the branch comparison on spend_before assumes that each member’s spend is independent of every other member’s, that spend within each level of the grouping variable is roughly normal, and that the two levels’ spreads are alike. In the summary table of an assumption check the Assumption column names the condition, the Status column carries the verdict, and the note beside it says what was tested and on what.

Status is the verdict column of a check: Pass or Fail, or a named verdict such as Equal variances, Sphericity assumed or MV normal, judged against the assumption test significance level – and a third state, Not assessed, when the check could not run. A Fail is a flag, not a sentence: the test recommendations under the table say what it costs and which test is safe regardless, and the note beside the status says what the test actually looked at, which matters because “normality” in one row means the values within each group and in another the residuals of a model. In a table that is not an assumption check the column names the row’s state in the procedure – selected or excluded, converged or not – and the manual for that table says which.

Not assessed is the status of a check that could not run – too few complete cases in a cell, a group with no variation, a model that could not be fitted – and it means unverified, not passed. The tests that rest on the assumption stay listed with a note saying so. The honest reading is that the assumption is unknown because the design or the sample is too thin to say, and a large n elsewhere on the card does not repair that.

Robustness is how far an assumption can be broken before a test’s error rates move: a test is robust to a violation when its real Type I error stays near the 5% it claims and its power near what it would have been. Robustness is a property of a test under a particular violation, not of a test. The t-test and ANOVA are robust to non-normality at moderate sample sizes, by the central limit theorem, and to unequal variances when the groups are the same size; Bartlett’s test and the F-test of variances are not robust to non-normality at any sample size; nothing is robust to dependence between the cases. The word is also used for robust statistics, which means something narrower – estimators that a few extreme values cannot move – and a robust test in that sense is one way of dropping an assumption rather than surviving its violation.

Whatever the field, the check is the same: a nurse’s ward comparison and a teacher’s class comparison assume independent patients or pupils, roughly normal outcomes within each group, and similar spreads across the groups, and the app’s assumption card asks those three questions in that order.

Independence #

Independence of observations is the assumption that one case’s value carries no information about another’s beyond what the analysis already accounts for – the foundation every test rests on and the one no test can check from the numbers alone, because it is a fact about how the data was gathered. The café’s 200 members are one row each, recruited and surveyed separately, so the members are independent, and that was settled when the survey was designed. Before and after are not independent: they are the same member measured twice, and a member who spent more than average before tends to spend more than average after (r = 0.69). Stacking the two till columns into 400 spends and running an independent-samples test on them gives Welch’s t(396.8) = 2.83, p = 0.005, with a 95% CI of [0.32, 1.76]; the paired test on the 199 within-member differences gives t(198) = 5.37, p < 0.001, [0.69, 1.49]. The estimate is +1.09 either way. The standard error is 0.37 in the stacked version and 0.20 in the paired one, because the pairing removes the spread between members (an SD of about 3.6 on either occasion) and keeps only the spread of the change within a member (SD 2.86). Ignoring a pairing costs power. The reverse mistake costs validity: a member surveyed at three visits and counted as three members, ten readings of one instrument counted as ten instruments, thirty pupils in one classroom counted as thirty independent trials of a teaching method – the sample is smaller than it looks, the standard error is understated, and the p-value is smaller than it should be by an amount no check will report. The dependent-samples designs, a subject ID and the table converter are how the app is told which rows belong together; what it cannot be told, it assumes.

Autocorrelation is dependence between cases that are neighbours in some order – successive days of a sales series, successive rows of a file sorted by time – and is independence’s usual failure when the rows are a sequence. Regression’s residual diagnostics test it with the Durbin–Watson statistic, near 2 when neighbouring residuals are unrelated, below 2 when each tends to resemble the one before, above 2 when they alternate, and the time series module is built around it. It is a question about an order, so the answer means something only when the row order is one. The café’s rows are members in the order the loyalty system listed them: the residuals of overall regressed on wait give DW = 2.34 in that order (p = 0.99 against positive autocorrelation) and 1.98 with the rows sorted by branch and then by waiting time – two answers to a question this data cannot be asked. When rows are genuinely ordered and the residuals track each other, the fix is a model that knows about the order – the regression module’s Newey–West standard errors, or a time-series model – not a test that ignores it.

Normality, briefly #

Normality has its own page; what belongs here is of what. A test on group means assumes the values are roughly normal within each group, and a regression assumes it of the residuals – never of the outcome pooled across groups, which is a mixture. Shapiro–Wilk on all 200 spend_before values gives W = 0.98, p = 0.013, and rejects; within Riverside it gives W = 0.990, p = 0.655, and within Station W = 0.985, p = 0.343, and the assumption the t-test actually makes holds. The pooled variable fails because it is two groups with different means and spreads laid on top of each other, which is not normal even when each part is. The app’s assumption card tests within cells for that reason, on the residuals for a covariate-adjusted or repeated-measures model, and adds a Q-Q plot of the pooled residuals beside the per-cell rows – and its recommendations read a normality failure beside the group sizes, since above the small-sample advisory floor the central limit theorem covers the tests on means and the card says so rather than demoting them.

Multivariate normality is the joint version of the assumption, made by the tests that use several dependent variables at once – MANOVA, discriminant analysis, and the estimators behind factor analysis and structural models. It asks that the variables be normal together: every one of them normal, and their scatter an ellipse with no bends or bulges, so that every weighted sum of them is normal too. Each variable being normal on its own is necessary and not sufficient, which is why a separate check exists. Within Station, spend_before is normal and wait is not (W = 0.917, p < 0.001), and the pair cannot be jointly normal; within Riverside, both are close enough and the pair passes.

Mardia’s test checks multivariate normality through two statistics: a multivariate skewness, read against a χ² with $p(p+1)(p+2)/6$ degrees of freedom for p variables, and a multivariate kurtosis, read as a z. Either rejecting fails the check. On spend_before and wait within Riverside (n = 95) the skewness is χ²(4) = 7.10, p = 0.131, and the kurtosis z = −0.46, p = 0.644 – MV normal; within Station (n = 90) χ²(4) = 30.74, p < 0.001, and z = 2.70, p = 0.007 – MV non-normal, and the reason is wait’s long tail, which is mostly Station’s. The three food items by branch pass in Riverside (p = 0.278 and 0.227) and fail in Station on the kurtosis (p = 0.037): five-point items can never be normal, and a MANOVA on them leans on the robustness of Pillai’s trace at these group sizes rather than on the assumption. The app runs the test per group, and on the covariate-adjusted residuals when a covariate is present, so the row checks the assumption the model makes.

Equal variances #

Homogeneity of variance – also written homogeneity of variances or homoscedasticity across groups – is the assumption that the groups’ spreads are the same in the population: the dependent variable varies as much within one level of the grouping variable as within another. Student’s t-test, classical one-way and factorial ANOVA, and the post-hoc tests built on a pooled error term (Tukey’s among them) assume it, because they estimate one common variance from all the groups together and use it for every comparison. On spend_before by branch the standard deviations are 2.97 at Riverside and 4.05 at Station – Station’s members vary more, as the codebook says the data was built to make them – and Levene’s test gives F(1, 198) = 9.10, p = 0.003: the summary row reads Unequal variances. On age the SDs are 9.88 and 10.11, F = 0.18, p = 0.669, and the row reads Equal variances.

Equal variances is the status Levene’s test prints when it finds no evidence that the groups’ spreads differ – see homogeneity of variance. It is absence of evidence: with two small groups the test has little power, so the verdict is read beside the standard deviations themselves, and a variance ratio of three on ten cases a side is unequal whatever the p-value says.

Unequal variances – the assumption’s failure, heteroscedasticity between groups – costs something that depends on the group sizes, not on the ratio alone. When the groups are the same size a pooled test barely notices: on spend_before, 101 members against 99 with a variance ratio of 1.86, Student’s t(198) = −3.21, p = 0.0015, against Welch’s t(179.5) = −3.20, p = 0.0016. When the sizes differ, the pooled variance is weighted toward the larger group and the error runs one way or the other. If the larger group also has the larger spread, the pooled standard error is too big and the test is conservative; if the smaller group has the larger spread, the pooled standard error is too small and the test rejects too often, which is the dangerous direction. The café has the first case: visits_before by recommend sets 135 members who would recommend the café (SD 6.13) against 65 who would not (SD 3.57), a variance ratio of 2.94, and Student’s t(198) = 2.16, p = 0.032, with a standard error of 0.82, where Welch’s t(190.8) = 2.57, p = 0.011, with 0.69 – the pooled test gives away power. Swap the spreads and the same imbalance would make Student’s t claim more than the data supports. Welch’s correction removes the assumption at no cost when it holds, which is why the app prefers it; for the rank tests, unequal spreads break the location-shift reading instead, and the recommendation moves to Brunner–Munzel.

Variance ratio is the larger group variance divided by the smaller – or the SD ratio, its square root – and the quickest read on how unequal the spreads are. The rule of thumb for the tests that pool: with similar group sizes a variance ratio up to about four, standard deviations within a factor of two, leaves an ANOVA’s error rate close to nominal, and the balance of the groups matters more than the ratio. spend_before by branch is 1.86 (SDs 2.97 and 4.05), wait 2.21 (2.30 and 3.42), visits_before 2.41, age 1.05. The F-test for equality of variances reports the ratio as its effect size with a confidence interval – 0.54 [0.36, 0.80] for Riverside over Station on spend_before, so Station’s variance is 1.86 times Riverside’s and the interval excludes 1 – and the comparison card’s Spread of the cell standard deviations table gives the largest-over-smallest SD across the cells of a design when an aligned-rank test is in play.

Levene’s test tests homogeneity of variance by turning it into a test of means: each value is replaced by its absolute distance from its group’s centre, and a one-way ANOVA on those distances asks whether the groups’ average distances differ – F(1, 198) = 9.10, p = 0.003 on spend_before by branch, F(1, 183) = 5.53, p = 0.020 on wait. The original version centres on the group means; the Brown–Forsythe version centres on the medians, and is the one the app runs wherever it checks variances – the comparison card’s Homogeneity of variance (Levene’s test, Brown-Forsythe) rows, and the correlation card’s Equal group variances check behind the point-biserial and η² methods. The centre matters on skewed or discrete data: taste by branch gives F = 2.14, p = 0.145, centred on the medians and F = 4.32, p = 0.039, centred on the means, from five-point answers whose medians sit on a level and whose means sit between two. Because it is an ANOVA on distances rather than a formula about the variances, Levene’s test is robust to non-normality in a way Bartlett’s test and the F-test are not, which is what qualifies it as the gatekeeper for the location tests.

Brown–Forsythe test is Levene’s test centred on the group medians rather than the means – the variant recommended whenever the groups may be skewed, and the app’s default. The name also belongs to the Brown–Forsythe one-way ANOVA (F*), a different procedure: a test of means that, like Welch’s ANOVA, refuses to pool the variances. The first is a check on the variances; the second is what to run when the check fails.

Bartlett’s test compares the group variances through their logarithms against the pooled variance, giving a χ² with one fewer degree of freedom than there are groups; K² = 9.42, p = 0.002 on spend_before by branch. It is the most powerful of the variance tests when every group is normal and the least trustworthy when one is not: a heavy tail inflates it at any sample size, with no central-limit rescue, because a variance follows an extreme value far more steeply than a mean does. On wait by branch (skewness 1.26) Bartlett’s K² = 14.03, p = 0.0002, where Levene’s gives p = 0.020 and Fligner–Killeen’s p = 0.026; on the near-normal spend_before all three land at p ≈ 0.002–0.003. The comparison module offers it as a test of spread in its own right and its assumption check demotes it when normality fails. Not to be confused with Bartlett’s test of sphericity in factor analysis, which asks whether the variables are correlated at all.

F-test for equality of variances is the two-group case: the ratio of the two sample variances, read against the F distribution with n₁ − 1 and n₂ − 1 degrees of freedom – F(100, 98) = 0.54, p = 0.002 on spend_before, Riverside over Station, with the variance ratio and its interval as the effect size. It assumes normality more strictly than any other test on this page and inherits Bartlett’s sensitivity to tails, and it can be run one-sided – “is Station the more variable branch?” – which the omnibus tests cannot.

Fligner–Killeen test is the rank-based variance test: the distances of the values from their group medians are ranked, the ranks are scored against normal quantiles, and the groups’ mean scores are compared as a χ² – 9.03, p = 0.003 on spend_before, 4.93, p = 0.026 on wait. It is robust to non-normality the way every rank test is, which makes it the dependable choice on skewed data, and it is the test the assumption check steers to when the F-test and Bartlett’s are demoted.

Welch’s correction – the Welch–Satterthwaite adjustment – is the fix for unequal variances in the tests on means: each group’s variance enters the standard error separately instead of through a pooled estimate, and the degrees of freedom are reduced to match the precision that was lost. On spend_before the t-test’s df fall from 198 to 179.5; they fall further as the variances and the sizes grow more unequal, down to one less than the smaller group’s size at the limit. Welch’s t-test, Welch’s one-way ANOVA, the Games–Howell and Dunnett’s T3 post-hoc tests and the Brown–Forsythe F* are all versions of it. When the variances and the sizes are equal it changes nothing – age by branch gives t(192.4) under Welch and t(193) under Student, to the same p – so the correction is free where it is not needed and right where it is, and the app’s test list puts the Welch version first.

The spread question travels: whether two wards’ recovery times vary alike, whether two classes’ scores do, whether two production lines’ tolerances do – the grouping variable’s levels, the dependent variable’s spread within each, and Levene’s test on the distances.

Along a line: homoscedasticity and linearity #

Homoscedasticity is the same equal-spread assumption made along a continuous predictor instead of across groups: in a regression – and behind the p-value and interval of Pearson’s correlation – the errors around the line are as spread out at one end of the predictor as at the other, so the residuals form an even band. The slope is estimated correctly either way; what depends on the assumption is the slope’s standard error, and with it the t, the p-value and the interval. Regressing spend_after on spend_before gives residual standard deviations of 2.48, 2.68 and 2.86 across the lower, middle and upper third of the fitted values – a gentle widening, with a Breusch–Pagan statistic of 3.69, p = 0.055, on the edge of the check’s threshold.

Heteroscedasticity is its failure – the band becomes a funnel, or narrows – and the usual causes are a bounded scale, a count, or a relationship that is multiplicative rather than additive. Regressing overall on taste gives residual SDs of 1.86, 1.77 and 1.23 across the thirds: the members who rated the taste highest are crowded against the ceiling of the 0–10 satisfaction scale, so the errors there are squeezed, and the Breusch–Pagan test gives BP = 4.26, p = 0.039. The remedy in the app is the regression module’s Standard errors selector: the HC3 heteroscedasticity-consistent robust standard errors re-estimate the precision without assuming the spread is constant, and the coefficient itself is untouched – the slope stays 0.878, its standard error goes from 0.097 to 0.108 and its t from 9.01 to 8.10, still far past any threshold. The correlation card reports the same test as its Homoscedasticity (Breusch-Pagan) column and, when it fails, keeps Pearson’s r and advises a bootstrap interval for it, since the coefficient is sound and only its analytic interval is not.

Linearity is the assumption that a relationship a model fits as a straight line is one, and it is the assumption that bites the coefficient rather than its precision: a curve fitted by a line gives a slope that is an average which holds nowhere on the range, and a Pearson’s r that understates the association. Ramsey’s RESET test, which both the correlation and the regression checks run, adds the squared and cubed fitted values to the model and asks whether they explain anything the line missed. spend_before regressed on age has a slope of 0.08 per year, p = 0.001, and RESET F = 5.89, p = 0.003: the relationship bends, because the members are two age groups – students and office workers – rather than a continuum, and a straight line through them describes neither. overall on wait gives RESET p = 0.11 and the line stands. When linearity fails, a rank correlation captures any monotonic relationship, a transformation may straighten the curve, and a model with a curve in it – a polynomial term, a spline – is the regression answer.

Sphericity #

Sphericity is the assumption of a repeated-measures design with three or more conditions: that the variances of the differences between every pair of conditions are equal. Two conditions have one difference and nothing to compare it with, so a paired t-test has no sphericity to assume; three conditions have three pairwise differences, six have fifteen, and repeated-measures ANOVA assumes they all vary alike because it pools them into one error term. The café’s members each rated three food items – taste, fresh and value – and “do members rate the three items differently?” is a within-subjects question with the items as the conditions (the file is wide, so the table converter stacks the three columns first). The variances of the pairwise differences are 1.34, 1.54 and 1.73, a ratio of 1.29; across all six items, with slow reversed, they run from 1.34 to 2.93, a ratio of 2.19. When sphericity is violated the F-test’s degrees of freedom are too generous and it rejects too often, and the two corrections below shrink the df to compensate. The word has a second meaning in Bartlett’s test of sphericity (factor analysis), where it describes a correlation matrix with nothing in it – a different assumption of a different model.

Mauchly’s test tests sphericity. Its statistic W is 1 when the pairwise-difference variances are exactly equal and falls as they spread, with a χ² approximation for the p-value. On the three food items W = 0.979, p = 0.119, and the row reads Sphericity assumed; on the six items W = 0.676, p < 0.001, Sphericity violated. The test has little power at small samples, where a real violation goes undetected, and over-rejects at large ones, where a trivial departure reads as significant, which is why the app applies the corrections to every within-subjects effect whatever Mauchly’s verdict and prints its table as information: with 200 members the six-item rejection is real, and the variance ratio of 2.19 says so independently.

Greenhouse–Geisser correction multiplies both degrees of freedom of the repeated-measures F by an estimate ε̂ of how far the pairwise-difference variances depart from equality – 1 when sphericity holds, down to $1/(k-1)$ at worst for k conditions (0.5 with three, 0.2 with six). The F itself does not change; its reference distribution does, and the p-value grows. On the six items F(5, 995) = 4.73 gives p = 0.0003 with sphericity assumed; with ε̂ = 0.838 the df become 4.19 and 833.7 and p = 0.0007. On the three food items ε̂ = 0.979 and the correction moves p from 0.016 to 0.017. The Greenhouse–Geisser estimate is known to run low, especially when the truth is near 1, so its correction is the conservative one.

Huynh–Feldt correction applies the same adjustment with a less biased estimate of ε that runs a little higher and is capped at 1: 0.858 on the six items, giving p = 0.0006, and 0.989 on the three food items. The choice between the two follows Girden’s rule – Greenhouse–Geisser when its ε̂ is below 0.75, Huynh–Feldt from 0.75 up – and the app reads its significance verdict from whichever that rule picks, tagging it GG-corrected or HF-corrected on the card so the driving p-value is never in doubt. Where the sphericity machinery is not wanted at all, the multivariate route tests the within-subject contrasts jointly and assumes no sphericity.

The columns a repeated-measures card carries for this:

• ε (GG) – the Greenhouse–Geisser estimate of ε; 1 means the pairwise-difference variances are equal and the correction changes nothing.
• p (GG) – the p-value of the same F read with both degrees of freedom multiplied by ε (GG).
• df (GG) – those corrected degrees of freedom, effect and error – 4.2 and 833.7 for the six items, against 5 and 995 uncorrected.
• ε (HF) – the Huynh–Feldt estimate of ε, a little larger than ε (GG) and never above 1.
• p (HF) – the p-value under the Huynh–Feldt correction, the one the verdict reads when ε (GG) is 0.75 or more.
• df (HF) – the degrees of freedom under the Huynh–Feldt correction – 4.3 and 853.9 for the six items.

Several outcomes at once: covariance matrices #

Homogeneity of covariance matrices is homogeneity of variance for several dependent variables at once: each group’s covariance matrix – the variances of the dependent variables and the covariances between them – is the same in the population. MANOVA, MANCOVA and discriminant analysis assume it, because they pool the groups’ matrices into one. spend_before and wait by branch have SDs of 3.00 and 2.30 at Riverside against 4.04 and 3.42 at Station, with the two variables correlated −0.04 in both – the matrices differ in the variances and not in the correlation. The three food items have SDs of 1.10, 1.07 and 1.06 at Riverside against 1.28, 1.22 and 1.35 at Station, and inter-item correlations of 0.58, 0.52 and 0.43 against 0.45, 0.42 and 0.35: Station’s members vary more on every item and their answers hang together less.

Box’s M test is its test, read as an approximate χ²: χ²(3) = 22.0, p < 0.001 for spend_before and wait by branch, χ²(6) = 17.1, p = 0.009 for the three food items, χ²(21) = 53.0, p = 0.0001 for all six. It is sensitive to non-normality and over-rejects at large samples, so by convention it is read at α = 0.001 rather than at the usual level: the food items pass at that threshold and the spend–wait pair fails. What a failure costs depends on balance, as it does for the univariate case. With groups of similar size, Pillai’s trace – the MANOVA statistic the app reports – absorbs a good deal of covariance heterogeneity, and the app keeps the MANOVA family recommended with a caveat when the largest cell is no more than one and a half times the smallest; with unequal cells it demotes them, since a MANOVA with the smaller group the more variable one rejects too often. The MANOVA section of the comparison manual says what the card shows in each case.

Covariates #

A covariate – a continuous variable added to a comparison to adjust for it, as in ANCOVA – brings two assumptions of its own, and the app’s assumption card checks both, along with the linearity of the covariate’s relationship to the outcome within each group.

Homogeneity of regression slopes is the assumption that the covariate’s relationship to the dependent variable is the same in every group – parallel lines, one per group – because ANCOVA adjusts every group by a single slope. It is checked by adding the covariate × group interaction and testing it. Adjusting spend_after by branch for spend_before, the slope of after on before is 0.74 at Riverside and 0.67 at Station, and the interaction gives F(1, 195) = 0.36, p = 0.549: parallel enough, and the single adjusted difference means something. When the slopes do differ, one adjusted difference does not exist – the branches differ by one amount among low spenders and another among high – and the app’s heterogeneous-slopes results report where along the covariate the groups differ instead.

Covariate independence – from the treatment, in the app’s fuller label – is the assumption that the covariate does not itself differ between the groups. When it does, adjusting for it removes part of the group difference along with the noise, and the adjusted comparison answers a different question from the raw one. spend_before differs by branch (7.91 against 9.43 among the 199 members with a valid after-spend, t = −3.05, p = 0.003), so the raw branch difference in spend_after of 1.63 (p = 0.002) becomes 0.56 (p = 0.147) once spend_before is adjusted for. Neither number is wrong. The adjusted one asks whether members who spent the same before the menu change differ in what they spend after – a fair question about the menu – and the raw one asks whether the branches differ, which they do, mostly because their members already did. age, by contrast, does not differ by branch (37.0 against 39.5 years, p = 0.080), and adjusting for it leaves the branch difference at 1.45 (p = 0.006): a covariate independent of the groups only removes noise. In an experiment with random assignment the covariate is independent of the groups by design and the adjustment is pure gain; in an observational comparison like this one, the assumption is the whole argument, and the report has to say which question the adjusted difference answers.

Categorical designs #

The tests on counts have no distribution to be normal, and their assumptions are about the counts being large enough for the χ² approximation and about the pairs being informative.

Expected frequencies are the counts each cell of a contingency table would hold if the two variables were independent – $\text{row total}\times \text{column total}/N$ – and the χ² test compares the observed counts against them. Its approximation needs those expected counts to be reasonable; Cochran’s rule, which the app applies, is that no cell should expect fewer than 1 and no more than a fifth of the cells fewer than 5. recommend by branch expects 68.2, 32.8, 66.8 and 32.2 against observed counts of 73, 28, 62 and 37 – comfortably safe. Cross recommend with whether the member waited more than ten minutes instead, and only 11 members did: the expected counts in that row are 7.4 and 3.6, one cell in four is below 5, and the χ² p-value is not to be trusted where Fisher’s exact test (p = 0.75) is. When the rule fails the app’s goodness-of-fit test switches to a Monte Carlo p-value on its own and says so under its table; the independence test warns under its table and leaves the switch to Fisher’s exact test to you.

Symmetry is the assumption of the Wilcoxon signed-rank test – paired and one-sample – that the differences are distributed symmetrically about their centre, since the test ranks the sizes of the positive and the negative differences on one scale. It is weaker than normality: a symmetric heavy-tailed sample fails a normality check and is still fine for the signed-rank test. The before-to-after change in spend has a skewness of −0.13, with a mean of 1.09 and a median of 0.93 – symmetric enough; the change in visits has a skewness of 0.69, a mean of 0.64 and a median of 0, because a few regulars added many visits and most members added none. The app checks it with a sign-flip permutation test on the skewness, and when it fails steers to the sign test, which assumes nothing about shape and counts only directions.

Discordant pairs are the pairs in a paired categorical design whose two conditions disagree – the member who would have recommended the café before and would not after, or the reverse. McNemar’s test and the Stuart–Maxwell test are built on those pairs alone; the concordant ones carry no information about change. The app’s check counts them and prefers the exact binomial test below 25, where the χ² approximation is coarse.

Informative subjects are the analogue for three or more paired conditions under Cochran’s Q: the subjects whose response varies across the conditions. A subject who answered the same way every time contributes nothing, and the approximation needs at least four informative subjects with their number times the condition count above 24.

When an assumption fails #

A failed check is a decision, not a verdict, and there is an order to take the options in:

1. Ask whether it matters. The recommendations under the card already do this: normality failed but every group clears the small-sample floor – the test on means is robust and stays recommended; Box’s M failed but the cells are balanced – Pillai’s trace stays recommended with a caveat. Above a few dozen cases per group, a normality rejection is usually a fact about the test’s power rather than about the analysis.
2. Use the test that does not assume it. Welch’s for unequal variances, the Brunner–Munzel test where a rank test’s shift reading breaks, Fligner–Killeen where Bartlett’s fails, the multivariate route where sphericity is in doubt, an exact test where the expected counts are thin – the parametric and non-parametric page walks the choice.
3. Correct rather than switch. The Greenhouse–Geisser and Huynh–Feldt corrections for sphericity and the HC3 standard errors for heteroscedasticity keep the model and repair its p-values; the app applies the sphericity corrections on its own, and HC3 is a choice on the regression module’s Standard errors selector.
4. Transform, with care. A log or a square root can equalise spreads and straighten a curve at once, at the price of an estimate on the transformed scale – the distributions page shows what the transforms do to wait.
5. Change the question or the model. A covariate that differs between the groups, a dependence between rows the design cannot express, a curve the line cannot follow: these are not repaired by another test, and the honest step is a different analysis – a mixed model with a subject ID, a regression with a curve, or a comparison that states plainly which question it answers.

What is never the right move is running every alternative and reporting the one whose p-value came out smallest; the check is there so that the choice of test can be made before the result is known, and defended afterwards.

Reporting #

Name the assumptions that matter for the test, say how each was checked and what was found, and let the test choice follow: “Welch’s t-test was used, as the branches’ spreads differed (Levene’s test p = 0.003)”; “sphericity was violated (Mauchly’s W = 0.68, p < 0.001), so the Greenhouse–Geisser correction is reported (ε = 0.84)”; “Box’s M was significant at α = 0.001, and with balanced groups Pillai’s trace is reported”. Independence is reported as a statement about the design – one row per member, before and after paired by member – because nothing on a results card can vouch for it. A covariate needs a sentence on both of its assumptions. And a check that could not run is reported as not assessed, with the reason.

Where this appears in DataSuite #

• Comparison analysis – the Check assumptions button and its card: normality and outliers per cell, Mardia’s test, Levene’s test, Mauchly’s test with both corrections, Box’s M, the covariate checks, expected frequencies, discordant pairs and informative subjects, and the test recommendations that read them; the tests of spread; the within-subjects approach; the heterogeneous-slopes results; Welch’s tests and the Brown–Forsythe F*.
• Correlation analysis – the per-pair suitability matrix: bivariate normality, RESET, Breusch–Pagan, influential points, equal group variances, expected counts, and which methods each failure drops.
• Regression analysis – normality of the residuals, Durbin–Watson and Breusch–Godfrey for autocorrelation, Breusch–Pagan for heteroscedasticity, RESET for linearity, and the robust standard errors that answer the last two; collinearity diagnostics for predictors that overlap.
• Factor analysis – Bartlett’s test of sphericity and the KMO measure, the other sense of the word.
• Structural equation modeling and confirmatory factor analysis – multivariate normality among the pre-fit diagnostics, and the robust estimators for when it fails.
• Time series analysis – autocorrelation as the thing modelled rather than the thing assumed away.
• Data transformation – the table converter that turns a wide file into the long one a within-subjects design needs, and the transforms that equalise spreads.
• Settings – the assumption test significance level and the advisory thresholds the recommendations are judged against.