Comparison designs #

A comparison asks whether a dependent variable differs between the levels of a factor, and a design is the arrangement of the factors: how many there are, whether each member of the sample sits in one level or is measured in all of them, and whether a covariate rides along. The guide sorts designs into the four the app’s Analysis type offers – independent, dependent, mixed and one sample – and its decision table picks a test; this page is the vocabulary those tests print once they run: main effects and interactions, cells and marginal means, the two routes through a repeated-measures test, the post-hoc families and the adjusted mean. In the app’s roles a design is spelled out in the left panel – a grouping variable is a between-subjects factor, a condition variable a within-subjects factor paired with a subject ID, every unassigned variable a dependent variable, and a covariate a continuous variable held fixed. The numbers come from the café data, prepared as that page describes; the crossed designs use the members’ spend before and after the menu change by branch, and three age bands – up to 30, 31 to 45, 46 and over, a binning with cut points at 31 and 46 that the power page introduces – with 47, 102 and 46 members. The second half of the page is the test families the guide’s decision table does not reach and the module’s test list does: several dependent variables at once, ranks in a design, decomposing a table, stratified tables, ordered alternatives, whole-distribution tests, robust comparisons, spread as the hypothesis, one variable against a number and paired categorical outcomes – each family’s blocks naming its tests and what they ask, with the manual holding the options.

Six neighbouring pages hold what this one leans on and does not repeat. Regression basics has the interaction as a product term, the covariate, adjusting for a covariate, the ANOVA table, the type of sums of squares and the Johnson–Neyman region of significance in the regression reading. Assumptions has sphericity and its corrections, the covariance matrices of a multivariate design and the two covariate assumptions. Hypothesis testing has the family-wise error rate and the adjustment methods a post-hoc table can fall back on. Effect sizes has η², partial η² and ω², the measures a design’s F rows carry, and the odds ratio, Cohen’s w and Cohen’s h of the categorical tests. Parametric and non-parametric has the trimmed mean and the permutation and exact routes to a p-value the robust and whole-distribution tests take. Association has entropy and mutual information, the quantities the information-theoretic tests are built on.

Factors and cells #

Factor is a categorical variable whose levels are the groups or conditions a comparison sets against each other – branch, with Riverside and Station; the occasion, with before and after. In the app a factor is a grouping variable or a condition variable, and a design is named by how many factors it has and of which kind: one-way for one, factorial for two or more crossed, mixed for one of each kind. It is not the factor of factor analysis – that is a latent variable inferred from the data; this one is a column you can see.

Level is one value of a factor – Riverside is a level of branch, before a level of the occasion. The number of levels sets the shape of the test: two levels make a t-test, three or more an ANOVA, and every post-hoc table is a list of level pairs.

Cell is one combination of levels across the factors of a design – Riverside members up to 30, Station members after the menu change – and the unit a factorial test compares, so two branches by three age bands make six cells, and the app’s groups preview counts them once a native factorial test is picked. On spend_before the cells hold 30, 50 and 19 members at Riverside for the three bands and 17, 52 and 27 at Station, with means of 6.53, 8.00 and 9.42 against 7.42, 10.11 and 9.52. Everything a factorial reports – main effects, interaction, estimated marginal means – is arithmetic on these six numbers and their spreads.

Between-subjects factor is a factor whose levels contain different members, each member in exactly one level. branch is one: nobody belongs to both. In the app it is a grouping variable, and the independent-samples designs are made of these alone; a between-subjects effect is tested against the spread between members, which is large, since members differ from one another in everything the study did not measure.

Within-subjects factor is a factor whose levels each contain the same members, measured once per level – before and after the menu change, every member spending in both. In the app it is a condition variable paired with a subject ID that says which rows are one member, and the file must be long, one row per member per level. What makes it worth the trouble is what the guide says: each member is their own control, so the differences between members leave the error term, and a within-subjects effect is tested against the spread of the changes alone. Of the 5,344 units of squared spread the café’s mixed model partitions, 4,173 sit between members and 808 within them.

Full crossing is a design in which every level of one factor meets every level of another – the six cells above, each occupied. The app builds it two ways, and they answer different questions. A native factorial test – factorial ANOVA, repeated measures factorial ANOVA, mixed ANOVA and their aligned-rank forms – fits the crossing as a decomposition: a main effect per factor and an interaction, each with its own F. The Combine variables into a single factor strategy, offered when a single-factor test is chosen with two or more factors selected, glues the cells into one factor of six levels and runs the single-factor test on it – one omnibus over the cells, with no term saying whether the branch gap and the age gradient are separate things or one thing. Use the strategy for a test that has no factorial form, a Kruskal-Wallis over six cells; use a native test when the question is about the factors.

Balanced design is a crossing in which every cell holds the same number of members. Its virtue is that the factors are then uncorrelated – knowing a member’s branch says nothing about their age band – so each effect is tested without reference to the others, the Type II and Type III sums of squares agree, and the marginal and estimated marginal means are the same numbers. The mixed design – 101 Riverside and 98 Station members, each measured twice – is as good as balanced.

Unbalanced design is a crossing whose cells differ in size, which is the ordinary state of observational data: 30, 50 and 19 against 17, 52 and 27 in the age-band crossing, because Station’s members are older. The factors are now correlated, the two sums-of-squares rules give different main-effect F’s, and the marginal and estimated marginal means part – the sections below say by how much on the café. An experiment assigns members to cells and stays balanced; a survey takes the cells it finds.

Main effect is the effect of one factor averaged over the levels of the others – does spend differ by branch, taking the three age bands together; does it differ by age band, taking the two branches together. On the factorial’s card each main effect is a row with its own F: on spend_before, age band F(2, 189) = 7.30, p < .001, partial η² = .072, and branch F(1, 189) = 3.76, p = .054, partial η² = .020, under the app’s default Type III sums of squares. A main effect is read from the estimated marginal means, not from the raw group means, and it is a fair summary only while the interaction is small – while the branch gap is much the same in every band.

Interaction in a comparison is the term asking whether one factor’s effect changes across the levels of another – the regression page’s interaction, read on cells instead of slopes. For branch × age band the branch gap is 0.89 among the youngest, 2.11 in the middle band and 0.10 among the oldest, and the interaction’s F(2, 189) = 1.49, p = .227, partial η² = .016, says three gaps that different are still compatible with one gap. In the mixed design the interaction is the question – did the branches change differently? – and F(1, 197) = 0.07, p = .79 says no: Riverside rose from 7.91 to 8.95 and Station from 9.43 to 10.57, one rise. The app’s interaction plot draws the cell means as one line per level of a factor across the levels of the other, and parallel lines are the picture of no interaction. When an interaction is present the main effects stop being the story, and the simple effects take over.

Simple effect is the effect of one factor within a single level of another – the branch gap among members aged 31 to 45, on its own. It is the follow-up to a significant interaction, which says the effect differs by level without saying where. The app’s post-hoc table for a factorial carries these as cell-pair contrasts under the interaction’s heading: Riverside 31 to 45 against Station 31 to 45 is −2.11, p = .023 under the exact multivariate t over the fifteen cell pairs, while the same gap in the oldest band is −0.10.

Marginal means are the plain means of a factor’s levels, each member counted once: 7.83 at Riverside over its 99 members with an age and 9.47 at Station over its 96; 6.85, 9.07 and 9.48 for the three bands. They are what a one-way test compares, what the summary statistics print, and what the members actually spent.

Estimated marginal means – adjusted means on the app’s card – are the level means a factorial model predicts, averaged over the other factor’s levels with equal weight: each branch’s is the average of its three cell means, whatever the cells’ sizes. For branch that is 7.98 and 9.01 against the marginal 7.83 and 9.47 – a gap of 1.03 rather than 1.64 – and the difference is the whole story of this crossing: Station’s members are older, older members spend more, so the raw branch gap is mostly an age gap, and the equal-weight average takes the age composition out. By age band the two hardly part – 6.97, 9.05 and 9.47 estimated against 6.85, 9.07 and 9.48 marginal – because the branches split each band about evenly. In a balanced design the two are identical. The post-hoc contrasts of a factorial are contrasts of the estimated means, and a one-way test’s of the marginal ones – which is why Tukey on the age bands gives 2.08 for the middle band against the youngest in the factorial and 2.22 in a one-way ANOVA on the same members. Under a covariate the same table holds the adjusted means.

Type II and Type III sums of squares are the two rules the app offers for what each factor’s F is tested against in a crossing – the regression page’s type of sums of squares, read for a comparison. Type III, the default, tests each term given every other, the interaction included, so a main effect is the test of the estimated marginal means: branch F(1, 189) = 3.76, p = .054. Type II tests a main effect given the other main effects with the interaction set aside, so branch’s row is the test of what branch adds over age band: F(1, 189) = 7.59, p = .006 – larger, because Type II does not hand the unbalanced cells’ overlap to an interaction the data do not support. The interaction’s own row is the same under both, as is everything in a balanced design. When the interaction is negligible Type II has the more power; when it is not, neither main effect means much on its own and Type III’s is at least a defined quantity.

Error term is the estimate of noise a test divides its effect by – the spread within the cells, as a mean square – and whether it is one number shared by every comparison or one number per comparison separates two families of tests. A pooled error term takes every cell’s spread as one estimate: one-way ANOVA and its Tukey and pooled-error pairwise t-tests refer every pair to the same mean square on N − k degrees of freedom, 192 for the three age bands, so a pair of small bands borrows precision from the large one. A separate error term takes each pair’s two spreads on their own, as Welch’s t does, on Welch’s fractional degrees of freedom: the Games-Howell pairs on the same bands run on 128.7, 80.7 and 92.0 df, and Welch’s one-way F is F(2, 104.6) = 12.41 where the pooled F is F(2, 192) = 8.42. Pooling is the more powerful when the spreads are equal and the wrong thing when they are not – the homogeneity of variance question – and it is why the app pairs the heteroscedastic omnibuses with post-hocs of their own rather than with Tukey. A within-subjects design has an error term per contrast, which is what retires the studentized range there.

Repeated measures and mixed designs #

Repeated measures is the design with a within-subjects factor: the same members measured under every level, one row per member per level. The paired t-test is its two-level case and repeated measures ANOVA the general one, with sphericity as the assumption a third level adds. What the design buys is in the café’s before-and-after spend: the members differ enormously from one another, and a between-subjects test of the two occasions would read the rise of about a unit against that spread; the repeated-measures test reads each member’s rise against the spread of the rises, and the occasion’s F(1, 197) = 28.74 is a t of 5.4. The price is the complete subjects rule and the sphericity machinery, and the app’s Dependent samples analysis type is where the design lives.

Mixed design is one between-subjects factor and one within-subjects factor together – the app’s Mixed model analysis type – and its card has three rows: the between main effect, the within main effect and their interaction, the last being the question the design exists for. Branch × occasion on spend, 199 members with both spends: branch F(1, 197) = 11.60, p < .001, partial η² = .056 – Station’s members spend more; occasion F(1, 197) = 28.74, p < .001, partial η² = .127 – spend rose after the menu change; the interaction F(1, 197) = 0.07, p = .79 – it rose by about a unit at each branch. The two main effects have different error terms: the between effect is tested against the spread between members, the within effect and the interaction against the spread of the changes within them, which is why one card prints two error sums of squares and why the within rows are the more sensitive. It is also why the occasion’s partial η² of .127 sits so far above its plain η² of .022: the partial measure takes the 4,173 units of between-member spread out of its denominator, the plain one leaves them in, and a report has to say which it quotes.

Complete subjects are the members with at least one row in every level of the within-subjects factor, and a repeated-measures or mixed test runs on them alone: a member with a missing occasion has no change to measure and is left out, while one with two rows in the same occasion is kept, its rows averaged into one value for that cell before the test. The app’s groups preview counts the complete members per condition variable, with a line for the members excluded for missing a condition and a line for those whose repeated rows will be averaged. The preview reads rows, not values: the café’s one member whose after-spend the data page blanks as a catering order still has both rows and is in the preview’s 200, and it is the test that drops that member for the blank, running on 199 – the N = 199 complete subjects the mixed card opens with. A large exclusion is a finding in its own right, since the members who dropped out may not be like the members who stayed.

Univariate route – Univariate (ε-corrected) in the app’s Within-subjects approach – is the classical repeated-measures F, one per within effect, with its degrees of freedom shrunk by the Greenhouse–Geisser or Huynh–Feldt ε when sphericity fails. It is the default and the more powerful route when sphericity roughly holds, and with two levels – before and after – there is nothing to correct, since one difference has no other difference to be unequal to.

Multivariate route – Multivariate (Pillai) – tests the within-subject contrasts jointly, as a MANOVA on the differences with Pillai’s trace as its statistic, and assumes no sphericity at all. It costs subjects – it needs more members than contrasts – and it loses power to the univariate route when sphericity is nearly true. The choice drives the whole card, the omnibus, the verdict, the effect size and the multiplicity pool, rather than adding a second table, and between-subjects effects are tested the univariate way whichever route is picked.

Within-subject confidence interval is an interval for a condition’s mean built from the spread of the changes rather than of the members, so that it shows what a repeated-measures test sees. The ordinary interval around the mean before, 8.66, runs from 8.16 to 9.16, and around the mean after, 9.75, from 9.23 to 10.26 (both over the 199 members with both spends, hence 8.66 rather than the 8.71 of all 200): they overlap generously, yet the paired test’s t is 5.4, because each member’s rise is measured against that member’s own baseline and the members’ differences from one another never enter. Cousineau’s construction removes each member’s average from their scores before the interval is taken, and Morey’s factor $\sqrt{k/(k-1)}$ undoes the shrinkage that step introduces; on the café the half-width falls from 0.50 and 0.52 to 0.28, and the intervals become 8.38 to 8.94 and 9.46 to 10.03. The app prints it as a second interval column per condition on a repeated-measures or mixed card and draws it on the error-bar plot. It does not cover the condition’s own population mean – for that the ordinary interval stands – and it says nothing about a between-subjects gap.

Difference-adjusted interval is the within-subject interval rescaled by $\sqrt{2}/2$, Baguley’s adjustment, so that two conditions’ bars overlap exactly when their difference is not significant at the chosen level: the reader’s instinct – the bars don’t overlap, so they differ – is wrong for ordinary intervals and right for these. On the café the half-width is 0.20, and the bars for before, 8.46 to 8.86, and after, 9.55 to 9.95, are 0.69 apart. It is the error-bar plot’s Difference-adjusted within-subject confidence interval option.

Post-hoc comparisons #

Post-hoc test is a test run after a significant omnibus to say which levels differ, with the family-wise error rate held across the whole set: an ANOVA’s F says the three age bands are not all alike and nothing more, and three pairwise t-tests at α = 0.05 would carry a one-in-seven chance of a false alarm among them. Every method below is a way of paying for the multiplicity – a reference distribution built for the set, or a p adjusted the way the adjustment methods do – and the app’s Post-hoc method list offers, for each test, the ones built on that test’s own assumptions, so a test with unequal spreads gets a separate-variance method and a rank test a rank method. A post-hoc contrast inherits its omnibus’s error term.

Pairwise comparisons are the post-hoc table’s rows: every pair of levels with the difference between them, its interval and its adjusted p. Three levels make three pairs, six make fifteen, and a factorial’s interaction adds every pair of cells. The app prints the table long, one row per pair, or as a matrix with the levels on both axes, and marks the significant rows the way every other table does.

Contrast is one comparison stated as a weighted combination of level means – a pair is the simplest contrast, +1 on one level and −1 on another – and the post-hoc table’s first column names it: 31 to 45 − up to 30, estimate 2.08. The word matters because a control-group comparison, a difference of differences inside an interaction and a trend across ordered levels are all contrasts, and the multivariate-t and Dunnett methods below are defined over a set of contrasts rather than over pairs.

All-pairwise comparisons compare every level with every other, $k(k-1)/2$ of them; many-to-one comparisons compare every level with one reference level alone, k − 1 of them. A smaller family is a smaller penalty, so a many-to-one method is the more powerful whenever the reference comparisons are the only ones wanted – each dose against placebo, each new page against the current one – and the app’s vs. control methods are its many-to-one forms.

Control group is the reference level of a many-to-one comparison – the level every other level is set against – chosen in the app under the post-hoc method, one select per factor the run fits, when a vs. control method is picked. The café has no natural control; a trial’s placebo arm, a school’s usual curriculum and a site’s current page are the pattern.

Simultaneous confidence interval is a set of intervals built so that all of them cover their true differences at once with the chosen confidence, rather than each on its own – the interval counterpart of an adjusted p. The intervals of a Tukey or multivariate-t table are simultaneous, which is why they are wider than a lone t-test’s would be: 0.63 to 3.53 for the middle band against the youngest, where an unadjusted interval on the same contrast would run from 0.87 to 3.29.

The families, keyed as the app’s Post-hoc method list spells them. Each is offered only with the tests whose premises it shares, so the list changes with the test:

• Tukey HSD – the all-pairwise method for a pooled-error design: every pair’s difference is referred to the studentized range, the distribution of the largest gap among k means drawn from one population, so the family-wise rate is exact for balanced cells and slightly conservative otherwise. The classic post-hoc of one-way and factorial ANOVA. On the age bands, averaged over branch: 31 to 45 against up to 30, 2.08, p = .0025; 46 and over against up to 30, 2.49, p = .0020; 31 to 45 against 46 and over, −0.41, p = .776.
• Pairwise contrasts with exact multivariate-t adjustment – the same contrasts as Tukey’s, calibrated on the exact joint distribution of their t statistics given the correlation they actually have, which the studentized range only approximates once the cells are unbalanced or the contrasts share no single error term – a within-subjects factor, a covariate. The app lists it first and makes it the default wherever a design offers both. On the near-balanced age bands it gives p = .774 where Tukey gives .776, and the gap grows with the imbalance. Its many-to-one form is Dunnett’s.
• Pairwise t-tests using pooled ANOVA error – each pair a t on the ANOVA’s pooled mean square and N − k df, with no multiplicity control of its own: the run-wide p-value adjustment is what corrects them, and with that setting at none they are the raw pairs. For a family you mean to adjust by a false-discovery-rate rule instead, or one a reviewer will adjust by hand.
• Games-Howell test – the all-pairwise method for unequal spreads: each pair a Welch t on its own two variances and its own Welch df, referred to the studentized range. The partner of Welch’s and the Brown-Forsythe one-way ANOVA, and the one the assumption check steers to when Levene’s fails. On the age bands the three pairs run on 128.7, 80.7 and 92.0 df, p = .0001, .0003 and .81.
• Dunnett’s test (vs. control) – the many-to-one method for a pooled-error design: k − 1 contrasts against the control, each a pooled-variance t, referred to the exact multivariate t over just those contrasts and the correlation they share through the control’s mean. More powerful than any all-pairwise method when the control contrasts are the only ones that matter.
• Dunnett’s T3 (vs. control) – Dunnett’s for unequal spreads: the same k − 1 contrasts, each a Welch t, referred to the studentized maximum modulus. The many-to-one partner of Games-Howell, offered with the heteroscedastic omnibuses.
• Dunn’s test – the all-pairwise post-hoc of the Kruskal-Wallis test: each pair a z on the difference in mean ranks, using the ranks of the whole sample rather than re-ranking the pair, so that the pairs stay on the omnibus’s scale; its p-values are adjusted by the run-wide setting.
• Dunn’s test (vs. control) – the same z restricted to the k − 1 control contrasts, the rank-based analogue of Dunnett’s, adjusted over those alone.
• Conover’s test – a post-hoc of the Friedman test: each pair of conditions a t on the difference in rank sums, with a standard error built from the ranks’ own spread. More powerful than Nemenyi’s and a little more liberal; adjusted by the run-wide setting.
• Nemenyi test – the other Friedman post-hoc: the rank-sum differences referred to the studentized range, Tukey’s calibration on ranks, carrying its own multiplicity control.
• Pairwise trimmed-means comparisons (lincon) – the post-hoc of the trimmed-means one-way ANOVA: each pair a Yuen t on the trimmed means at the same trim level as the omnibus, on Winsorized variances; its p-values follow the run-wide setting, and its intervals are simultaneous, on the studentized maximum modulus.
• Pairwise paired t-tests – for a repeated-measures design, each pair of conditions a paired t on the members’ differences, adjusted by the run-wide setting.
• Pairwise Wilcoxon signed-rank tests – the same on ranked differences, the Friedman test’s other post-hoc, adjusted by the run-wide setting.
• Pairwise McNemar tests – after Cochran’s Q on a yes/no outcome, each pair of occasions a McNemar test on the members who changed, adjusted by the run-wide setting.
• Pairwise Jensen-Shannon divergence tests – after the Jensen-Shannon divergence test on a categorical outcome, the divergence between each pair of groups’ distributions, each bounded by one bit so that every pair reads on the omnibus’s scale; adjusted by the run-wide setting.

Covariates in a comparison #

ANCOVA – analysis of covariance – is a group comparison with a continuous covariate in the model, so that the groups are compared at a common value of it. It is the regression page’s adjusting for a covariate in a comparison’s vocabulary, and it asks a different question from the raw comparison: not “do the branches differ in after-spend” but “do members who spent alike before differ after”. In the app a covariate is a role in the left panel, offered for the tests that admit one – ANCOVA and MANCOVA, the factorial, mixed and repeated-measures families – and the card gains a covariate row, the adjusted means and the slope checks. Two assumptions ride with it, homogeneity of regression slopes and covariate independence, and in an observational comparison the second is the whole argument.

Adjusted mean is a group’s mean predicted at the covariate’s overall mean – every member moved along the common slope to what they would have spent had their before-spend been the average 8.66. spend_after by branch: raw means 8.95 and 10.57, adjusted 9.47 and 10.03, because Station’s members spent more before, so their raw mean is moved down and Riverside’s up along a slope of .70 per unit. The branch effect goes with the gap: F(1, 196) = 2.12, p = .147, partial η² = .011, beside the covariate’s F(1, 196) = 165, partial η² = .457 – the assumption page’s 1.63 → 0.56 seen from the other side. The value the covariate is held at is printed under the table, and it matters: adjusted means at another value are other numbers, and the difference between them is the same only while the slopes are.

Heterogeneous slopes is the case ANCOVA’s assumption rules out – the covariate’s slope differs between the groups, so that the group difference depends on where along the covariate it is read and there is no single adjusted difference to report. The app tests it in the assumption check and, on the results card, prints a slopes table with each contrast’s slope difference; where the slopes do differ it draws the Johnson–Neyman region of significance for that contrast – the stretch of the covariate over which the groups differ – and the Show every region of significance option draws one for every contrast. On the café the slopes of after-spend on before-spend are .74 at Riverside and .67 at Station, a difference of .07 (SE .11), p = .549; no region is drawn unless asked, and the one that can be asked for is empty, since the branch difference is significant at no before-spend inside the observed range of 1.5 to 18.9.

Several dependent variables at once #

MANOVA – multivariate analysis of variance – compares the groups on several dependent variables together: not whether the branches differ in overall satisfaction, then in wait, then in spend, but whether they differ in the profile of the three. Two things are gained. The false-alarm rate is paid once, where three separate ANOVAs pay it three times; and a difference that lives only in the combination – no variable separates the groups on its own, the pattern does – is found at all. The cost is a pair of assumptions the univariate tests do not make, homogeneity of covariance matrices and multivariate normality, and a verdict that says only that the groups differ somehow on the set – the follow-up tests and the discriminant functions say how. On overall, wait and spend_before by branch – 93 Riverside and 85 Station members with all three answers, 22 rows dropped – the card’s four statistics agree on F(3, 174) = 6.28, p < .001, and the multivariate partial η² beside Pillai’s row is .098: the branches differ on the profile. In the app it is the MANOVA test of the independent-samples list, every unassigned numeric variable one of its dependent variables; with one dependent variable it is an ANOVA.

MANCOVA is MANOVA with a covariate held fixed – the branches compared on the three-variable profile at a common value of, say, age – and its card adds a multivariate test per covariate and the adjusted means of every dependent variable per group, with the value each covariate was held at noted beneath. The two covariate assumptions apply to every dependent variable at once. In the app it is MANCOVA (Multivariate ANCOVA).

A multivariate test summarises the separation along the discriminant functions – as many as the smaller of the number of dependent variables and the number of groups less one – and the four statistics on the card are four ways of adding those functions up. With two groups there is one function, the four are one number in four spellings and give exactly the same F; with three or more they part, and the age bands on the same three variables (173 members) show by how much: Pillai .095, F(6, 338) = 2.81, p = .011; Wilks .906, F(6, 336) = 2.84, p = .010; Hotelling–Lawley .103, F(6, 334) = 2.87, p = .010; Roy .095, F(3, 169) = 5.37, p = .001.

Pillai’s trace is the sum over the functions of the share of variance each explains – each function’s canonical correlation squared, so .098 for the branches and .087 + .008 = .095 for the age bands – and it grows with separation. It is the statistic that stands up best when the covariance matrices differ or the groups are small and unequal, the one the app reads its multivariate effect size from, and the one to report when the four disagree.

Wilks’ Λ is the share of variance the functions leave unexplained – the product over the functions of one minus each canonical r² – so it runs the other way, from 1 at no separation downwards: .902 for the branches, which is 1 − .098, and .906 for the age bands. The most commonly reported of the four, and the one the discriminant table tests function by function.

Hotelling–Lawley trace is the sum of the functions’ eigenvalues – for each function, the spread between the group means along it over the spread within the groups – .108 for the branches and .103 for the bands. It weights a strongly separating function more than Pillai’s does, and is less robust than Pillai’s to unequal covariance matrices.

Roy’s largest root is the first function’s eigenvalue alone – .108 and .095 – so it asks whether the groups differ along their single best line and ignores every other. Its F is an upper bound, exact only when there is one function, which makes its p a lower bound: on the age bands it reads p = .001 where the other three read .010, and the app’s note on the row says so. It is the most powerful of the four when the groups genuinely differ along one dimension and the most misleading when they do not.

Univariate follow-up tests are one test per dependent variable after a significant multivariate result – ANOVAs, ANCOVAs under a covariate, Welch’s ANOVAs under Johansen’s, energy tests under the energy test, the card’s heading naming which – to say which variables carry the difference. They are gated by the omnibus rather than pooled into the run-wide adjustment, and adjusted among themselves as a family of one test per variable by the adjustment method you set. On the branches: spend_before F(1, 176) = 12.30, p < .001, partial η² = .065; overall F(1, 176) = 1.41, p = .24; wait F(1, 176) = 2.31, p = .13 – the multivariate separation is the spend gap wearing a combination, and the standardized coefficients below say what the combination adds. On the age bands only spend_before follows up, F(2, 170) = 7.30, p < .001.

Johansen’s heteroscedastic MANOVA is the multivariate Welch’s correction: each group is weighted by its own covariance matrix instead of one pooled matrix, so the test stands when Box’s M fails – as it does on the branches, whose spend and wait vary more at Station. On the three variables by branch it gives a statistic of 18.64, F(3, 124.9) = 6.15, p < .001, with Welch’s ANOVAs as its follow-ups – spend_before F(1, 154.2) = 11.98, p < .001, overall p = .24, wait p = .14 – and its fractional degrees of freedom are the price of estimating a matrix per group. Three things follow from that price: every group needs more members than there are dependent variables, the omnibus carries no effect size, and there is no pooled term for a post-hoc, so its follow-ups pair with Games-Howell. It still assumes multivariate normality.

Energy test of equal distributions asks whether the groups share one joint distribution at all – location, spread and shape together – with no distribution assumed, by permutation: the energy distance between the groups is computed, the group labels are shuffled and the distance recomputed, and p is the share of shuffles that matched or beat the observed value. On wait and spend_before together – 95 and 90 members – E = 8.22, p = .001, the floor of 999 permutations, with R²ₑ = .026; its per-variable follow-ups, energy tests of their own with an E and no df, read E = 1.92, p = .125 on wait and E = 8.40, p = .001 on spend_before, so the joint verdict is spend’s. A rejection is not evidence that the means differ – a difference in spread alone rejects – which is why the app offers it as the fallback when multivariate normality fails and not as a MANOVA substitute, and why a failed Box’s M demotes it too.

Energy distance between two groups is twice the average distance between a member of one and a member of the other, minus the average distance within each group – zero when the two distributions are the same and positive otherwise, whatever the shape of the difference. The dependent variables are standardized first, since a variable in larger units would otherwise decide the distances on its own. R²ₑ is the share of the total dispersion that lies between the groups, read off the same distances – 2.6% on the café pair – and is the effect size the card prints beside E, bounded and comparable across datasets where E itself grows with the sample.

Descriptive discriminant analysis names the combination the multivariate test found: a discriminant function is the weighted sum of the dependent variables along which the groups are furthest apart relative to the spread within them, and the analysis reports the functions, their weights and where each group sits on them. It is the follow-up matched to the omnibus – it reads the variables together, as the test does – and in exchange it tests no single variable, which is why the app shows it beside the univariate follow-ups rather than instead of them: Include descriptive discriminant analysis, offered for MANOVA and MANCOVA. On the branches there is one function, canonical r = .31, Wilks’ Λ = .90, χ²(3) = 17.95, p < .001 – the omnibus restated. On the age bands there are two, and the table’s tests are sequential: the first row tests both functions together, Λ = .906, χ²(6) = 16.72, p = .010, and the second whether any separation is left once the first is removed, Λ = .992, χ²(2) = 1.33, p = .52 – the bands differ along one line, which carries 92% of the separation, and the picture the discriminant function plot draws is one-dimensional in all but name.

Discriminant function is one row of the first table: its eigenvalue – the spread between the group means along the function over the spread within the groups, .108 for the branches – its canonical correlation, its share of the total separation, and Wilks’ Λ with the χ², df and p that test it together with every function after it. Functions are extracted in order, each separating the groups as well as possible among the directions the earlier ones left, so the first is the line the groups differ on most and a later one is what remains.

Canonical correlation is the correlation between a function’s scores and group membership – for two groups, the point-biserial r between the score and the branch – and it is the eigenvalue rescaled to a correlation, $\sqrt{\lambda /(1+\lambda )}$: .31 for the branches, .30 and .09 for the two age-band functions. Its square is the function’s share of variance explained, which is what Pillai’s trace adds up.

Standardized canonical coefficients are the weights the function gives each dependent variable once the others are held constant, in pooled within-group standard deviations – what a function is made of. On the branches: spend_before .96, overall −.48, wait .27. The function is mostly spend, but not only: a member who is more satisfied than their spend predicts is pulled toward the Riverside end, and one who waited longer toward Station’s, and the two univariate follow-ups that were not significant are both in it. That is the separation the multivariate test found and the per-variable tests cannot see.

Structure coefficients are the pooled within-group correlations between each dependent variable and each function: spend_before .80, wait .35, overall −.27. They describe one variable at a time, so they rank the variables much as the univariate follow-ups do, and reading them for what a function is built from reproduces the blindness the analysis exists to fix; the app shows them second for that reason.

Group centroids are each group’s mean score on each function, in pooled within-group standard deviations, so a gap of 1 is one within-group SD of separation: Riverside −.31 and Station .34, 0.66 SD apart on the one function; on the age bands’ first function, up to 30 at −.53, 31 to 45 at .14 and 46 and over at .25 – the youngest members stand apart from the other two, which is the spend gradient the trend test below reads directly. Under MANCOVA the centroids are the adjusted means projected onto the functions.

Ranks in a design #

Aligned rank transform is the way to run a factorial or mixed design on ranks when the residuals are far from normal. Ranking the response outright and running the factorial ANOVA on the ranks does not work: ranks are not additive, so the main effects and the interaction bleed into one another and an interaction can turn up significant where there is none. Alignment fixes this one effect at a time – for each main effect and interaction, every other effect’s estimate is subtracted from each observation, leaving that effect and the noise; the aligned values are ranked; and the ordinary factorial ANOVA on those ranks reads that one effect. The app’s ART-ANOVA (Aligned Rank Transform) is the mixed design’s non-parametric partner and Repeated measures factorial ART-ANOVA the within-only one, each fitting one aligned-and-ranked response per effect. On branch × occasion on spend: branch F(1, 197) = 10.76, p = .001, occasion F(1, 197) = 24.92, p < .001, the interaction F(1, 197) = 0.13, p = .72 – the parametric mixed design’s verdicts, at 11.60, 28.74 and 0.07, from ranks – which is what should happen on near-normal data, and the transform earns its place on data that is not. What it costs is set by what it fits: one random intercept per subject, so there is one error term across every cell – no sphericity correction, and unequal cell variances make every F in the table anti-conservative, the between-subjects rows included, which is why the card prints the spread of the cell standard deviations; no covariates, since a continuous predictor has no levels to align against; no simple effects, since alignment removes exactly what a simple effect would read; and partial η² as its only effect size.

Decomposing a table #

Correspondence analysis takes the contingency table the chi-square test just examined and says where the association lives: it splits the table’s chi-square into independent dimensions and places every row and column category on them, so that categories lying the same way from the centre are ones that co-occur more than independence predicts. It is a description, not a test – it has no p, the test it accompanies has done that part – and it is opt-in in the app: Include correspondence analysis, offered with the chi-square and Fisher tests. A table has one dimension fewer than its smaller side, so a 2 × 3 table decomposes onto a single line and only a 3 × 3 or larger table draws a map. recommend by age band is the café’s case, χ²(2) = 0.54, p = .764 over 195 members: the oldest band sits at .077 on the one axis, the youngest at −.075 and the middle band at the origin, with no at .074 beside the oldest and yes at −.037 beside the youngest – the oldest members recommend least, 63% against 70%, an association too small to matter, laid out on a line. A satisfaction rating with five levels against four branches would fill a plane.

Inertia is the amount of association, in the units correspondence analysis works in: the table’s total inertia is its chi-square divided by the sample size – .0028 for recommend by age band, 0.54 ⁄ 195 – and each dimension carries a share of it, printed as a percentage and a running total, which is what decides whether a two-dimensional picture is worth reading. Each category has an inertia too, its share of the total association: no carries 67% and yes 33%, the rarer answer sitting further from the average profile; 46 and over carries 50.5%, up to 30 49.5% and 31 to 45 none, since it sits at the origin.

Mass is a category’s share of the table’s count – .33 for no and .67 for yes, .52, .24 and .24 for the three bands. A light category moves the solution little however far out it lies, and a heavy one anchors it, so the map is read with the masses in mind: an extreme position for a rare category is a fact about a few members.

Contribution is how much of a dimension a category built – the share of that dimension’s inertia that comes from it, read down the column and summing to 100% over the categories. A dimension is named by the few categories that contribute most to it: on the café’s one axis, no contributes 67% and yes 33%, the oldest band 50.5% and the youngest 49.5%.

Quality is how well a dimension represents a category – the squared cosine of the angle between the category’s position and the axis, read across the row and summing to 1 over all the dimensions – and Total quality is the same over the dimensions the map draws. A category with low quality is not described by the picture, however far from the origin it sits; contribution says what the category did for the axis, quality what the axis does for the category, and a category is read from the map only when both are high. On a one-dimensional table every category has quality 1, save the middle band at the origin, which the axis says nothing about.

Symmetric map is the plot’s scaling: rows and columns are both drawn in principal coordinates, so distances within the rows and within the columns are the chi-square distances between their profiles, but a distance between a row and a column is not one, and the map is read by direction from the origin – a row and a column lying the same way co-occur more than expected, opposite ways less, and a category near the centre is close to the average profile and distinguishes nothing. Each axis carries its dimension’s share of inertia, and the app’s Correspondence map drops the lightest labels where they would overlap and says how many.

Multiple correspondence analysis decomposes three or more categorical variables at once – what the variables share, rather than one table at a time – by running the analysis on the indicator coding of every category, and it is the All selected variables together choice under Decompose. The inertias it reports are adjusted for that coding, Greenacre’s correction, so that its percentages read like a two-way analysis’s and do not sum to 100. On branch, recommendation and age band – 195 members, seven categories – the adjusted total inertia is .013 and the first dimension holds 90% of it with the second holding nothing: one axis, with Station at .74, no at .73 and 46 and over at .81 on one side, Riverside at −.72, yes at −.36 and up to 30 at −1.04 on the other, and 31 to 45 near the centre at .12. That is the three facts the pages before this one found separately – Station’s members are older, older members recommend less and so does Station – as one direction, and the quality column keeps it honest: the axis describes the branches (.53) and the age bands (.58 to .97) far better than the recommendation (.29), so the recommendation’s place on it is the weakest reading.

Stratified tables #

Stratum is one level of a third variable inside which an association is examined on its own: the members aged 31 to 45, and within them the table of recommendation by branch. Stratifying holds the third variable constant the way a covariate holds a continuous one, and the app’s Stratifying variable role, offered for the Cochran-Mantel-Haenszel test, builds one contingency table per stratum and prints them under the pooled table.

Cochran-Mantel-Haenszel test tests the association between a categorical outcome and a grouping variable within the strata, pooling the evidence across them: in each stratum the count in one cell is set against what the stratum’s own margins predict, and the gaps are summed, so an association has to point the same way in every stratum to add up. recommend by branch stratified by age band, over the 195 members with a band: the pooled table has 71 of Riverside’s 99 members recommending (72%) against 59 of Station’s 96 (61%); inside the bands the gap is 77% against 59% among the youngest, 72% against 62% in the middle and 63% against 63% among the oldest – strata that agree with the margin, which is why stratifying changes the verdict little here: CMH χ²(1) = 1.59, p = .207, with the continuity correction the app applies by default. It works for any table shape, with (I − 1)(J − 1) degrees of freedom; the 2 × 2 × K case adds the common odds ratio and the Breslow-Day check.

Common odds ratio is one odds ratio summarising every stratum, each weighted by its size – the Mantel–Haenszel estimate, the 2 × 2 × K case’s effect size when effect sizes are on. On the café strata it is .65 [.35, 1.18], in the orientation the card’s note spells out – Riverside’s odds of a no relative to Station’s, or 1 ⁄ .65 = 1.54 for Riverside’s odds of a yes – against .63 in the pooled table and .43, .62 and .99 within the three bands, youngest to oldest. The interval contains 1, which is no association.

Breslow-Day test asks whether the strata’s odds ratios are alike enough for one common ratio to summarise them – the stratified analogue of an interaction. χ²(2) = 0.85, p = .652 on the café: .43, .62 and .99 are compatible with one ratio, and the common .65 is a fair summary. When it rejects, the association differs by stratum and the honest report is the per-stratum tables, not the pooled ratio. The app runs the Tarone-corrected version, whose reference distribution is the χ² the plain one only approximates.

Simpson’s paradox is an association in the pooled table that vanishes or reverses inside every stratum, because the strata differ in both the mix of groups and the rate of the outcome. The café shows the mild form: the pooled gap of ten points is 18 among the youngest, 10 in the middle band and 0 among the oldest, so most of the pooled association belongs to one band, and it does not reverse. The classic form is a hospital: the harder cases go to the better surgeon, whose pooled death rate is therefore the higher, while within every severity stratum it is the lower – and a pooled table would recommend the worse surgeon. Stratifying, and testing within the strata, is the protection; the stratifying variable has to be chosen from what is known about the data before the run, since the pooled table cannot say whether it is needed.

Ordered alternatives #

Trend test is a test whose alternative is an ordering of the groups rather than any difference among them – the outcome rises, or falls, along an ordered factor – and it spends its power on that one direction where a k-group omnibus spreads it over k − 1 directions, so it is the sharper question whenever the order is real. The app has three: the Jonckheere-Terpstra test for a numeric outcome across ordered groups, the rank test that counts how often a member of a lower level is beaten by a member of a higher one; Page’s trend test for ordered conditions within subjects, the Friedman family’s trend test; and the Cochran-Armitage trend test for a binary outcome’s rate across ordered groups. Each needs an ordinal factor, and the app reads the order from the factor’s numeric values – the age bands coded 1, 2 and 3 work, and labelled up to 30, 31 to 45 and 46 and over they do not, since text levels sort alphabetically and 31 to 45 would come first. spend_before across the bands so coded: J = 7,306.5, p = .002 – a permutation p over 999 replicates, since the spends carry ties – with Kendall’s τ-b = .20 as its effect size, on means of 6.85, 9.07 and 9.48. recommend across the same bands: Cochran-Armitage Z = −0.73, p = .463 on rates of 70%, 67% and 63%, against the 2 × 3 chi-square’s χ²(2) = 0.54, p = .764 – the sharper question on one degree of freedom, and no trend to find. Page’s test has no instance in the café, whose conditions are the two occasions and the six unordered items; a dose taken at three levels by every patient, or a rating repeated at three months, six and twelve, is its shape.

Monotone trend is what the tests above look for: a rise or a fall that never reverses, not necessarily a straight line and not necessarily evenly stepped. It is stated before the run, in the app’s Trend direction control – increasing, decreasing, or any monotonic trend, the two-sided form – the way a one-tailed test’s direction is; Page’s test is directional only, since it tests a specified ordering. Jonckheere-Terpstra and Page read only the order of the levels; Cochran-Armitage fits a straight line to the rates against the trend scores, so its trend is a linear one.

Trend scores are the numbers assigned to the ordered levels – the x of the line the Cochran-Armitage test fits – and the app takes them from the factor’s own values, so levels coded 1, 5 and 10 are tested at that spacing rather than as three equal steps. The card discloses the scores it used. A signed Z is the statistic, positive for a rising rate, and its square is the usual one-df trend χ²: 0.54 on the café’s recommendation.

Whole-distribution tests #

Most of the tests on this page ask about one feature of a distribution – its mean, its median, its spread. The tests here ask the broad question, whether two groups’ distributions differ at all, and the answer comes with a cost stated at the end of the section.

Two ECDFs together – one ECDF per group on the same axes, as the app’s ECDF plot draws them – show where two distributions part and how, which a pair of means and SDs cannot. A location shift is one whole curve displaced sideways from the other; a spread difference is one curve flatter than the other while the two cross near the middle; curves that cross more than once differ in shape – a floor, a ceiling or a cluster that one group has and the other lacks. The café’s waiting times by branch are the second kind: the two curves interleave below four and a half minutes, cross 50% two-tenths of a minute apart – medians 4.8 and 5.0 – and then come apart, Riverside’s climbing to 100% at 11.4 minutes while Station’s, flatter on the right, still has 4% of its members to come and reaches 19.9. A gap between the curves reads two ways, and the two peak in different places: a vertical gap is a difference in shares at one value – at 8.5 minutes 95% of Riverside’s waits are over and 80% of Station’s – and a horizontal gap is a difference in values at one share – nine members in ten are served within 7.6 minutes at Riverside and 9.8 at Station. The largest vertical gap is Kolmogorov-Smirnov D, which the plot marks at the value it falls at; the largest horizontal one is the largest quantile difference – here the top step, 11.4 against 19.9 minutes, where the curves are nearly flat and the vertical gap is almost nothing – which no test on the card reads. The confidence band around each curve is that group’s alone, and two pointwise bands that clear each other where the gap is widest prove less than they seem: at 8.5 minutes the Wilson bands run 0.88 to 0.98 and 0.71 to 0.87 and do not touch, yet D = .147 is not significant, p = .209, because the value was found by looking along the whole curve for the widest gap, which a pointwise band never allows for. The DKW bands, ±0.14 either side of each curve, still overlap there, and that curve-wide reading is the one the test agrees with.

Kolmogorov-Smirnov D is the largest vertical gap between two groups’ ECDFs, the statistic of the Kolmogorov-Smirnov test: D = .147 on wait by branch, p = .209 over 95 and 90 members, an exact p until the product of the two group sizes reaches 10,000. It reacts to a difference of any kind – location, spread or shape – and is most sensitive in the middle of the distribution, where the curves are steepest. The test is two-sided; a note on the card records that this is conservative for a directional reading.

D⁺ and D⁻ are the gap in each direction – the most the first group’s curve runs above the second’s, and the most it runs below – with the value of the variable at which each is reached: D⁺ = .147 at 8.5 minutes and D⁻ = .045 at 1.1 minutes on the waits. By 8.5 minutes, 14.7 percentage points more of Riverside’s members had been served than of Station’s, and nowhere does Station’s curve run much ahead: whatever separation there is sits among the long waits, in Station’s tail. D is the larger of the two, and the card’s Larger gap column names which.

Anderson-Darling k-sample test asks the same question with every gap counted rather than the largest, weighted towards the tails, for two or more groups at once: T = 0.75, p = .166 on the waits, a simulated p over 999 replicates because the waits carry ties. It is the more sensitive of the two when the difference lives in the extremes, and the one to use for a single verdict over several groups; Kolmogorov-Smirnov auto-expands to a pairwise table instead.

Information-theoretic tests measure, in bits, how much knowing the group tells you about the outcome – the mutual information between the group label and the outcome, estimated from each value’s nearest neighbours rather than by discretizing the outcome into bins – and test it by permutation. The Mutual information test reports the information itself, bounded by the entropy of the group variable, with Theil’s U as its effect size; the Jensen-Shannon divergence test reports a distance. Both react to a change of any kind, and the café says what that costs. spend_before by branch, where Welch’s t reads p = .002 on a shift of 1.6, gives I = 0.017 bits, U = .017, p = .336: the test spreads its sensitivity over differences of every kind, and a plain shift is only one of them. wait by branch, where the location tests find nothing – Mann-Whitney p = .289, Kolmogorov-Smirnov p = .209 – gives I = 0.087 bits, p = .047, and a divergence of 0.066 bits, p = .072: the branches’ waits differ in shape, Station’s long tail, more than in the middle. A test for any difference is the right tool when the question is “are these the same distribution?” and the wrong one when it is “which is higher”.

Jensen-Shannon divergence is the distance between distributions: the average of each group’s divergence from their mixture, symmetric, and bounded by 1 bit for two groups and log₂ k for k, so that its normalized form – the divergence over that bound – reads on one scale whatever the group count. The app’s test is an omnibus over all the groups, with a pairwise breakdown as its post-hoc.

Robust comparisons #

The parametric and non-parametric page defines the trimmed mean and the winsorized spread; the tests built on them are the comparison module’s answer to the corner where heavy tails and unequal spreads arrive together.

Yuen’s trimmed-means t-test compares two groups’ trimmed means – 20% off each tail at the app’s default trim level – with a standard error from the winsorized variances and a Welch-style adjustment for unequal spreads. spend_before by branch: trimmed means 7.79 and 9.33, difference −1.54 [−2.59, −0.49], t(109.7) = −2.91, p = .004, beside Welch’s t(179.5) = −3.20, p = .002 – the same finding with fewer degrees of freedom, since trimming leaves 61 members a side to estimate from. With three or more groups it auto-expands to a pairwise table.

Trimmed-means one-way ANOVA (t1way) is the k-group version, the robust counterpart of Welch’s ANOVA, with the lincon pairs as its post-hoc: spend_before across the age bands, F(2, 59.0) = 7.94, p < .001, with the explanatory measure ξ = .38 [.24, .57] as its effect size, beside Welch’s F(2, 104.6) = 12.41.

Brunner-Munzel test is the rank family’s robust member. The Mann-Whitney test’s “one group tends to be higher” reading assumes the two distributions share a shape and differ only by a shift; Brunner-Munzel drops that assumption, estimates the relative effect directly and gives it a standard error of its own, so it stays valid when the spreads or shapes differ. On the waits, W = 1.06, df = 169.8, p = .293, beside Mann-Whitney’s U = 3889, p = .289: the two agree because the branches’ waits are shaped alike, and they part when they are not, which is when the assumption check steers here. At small samples its p comes from the exact permutation distribution.

Relative effect – p̂ – is the probability that a randomly drawn member of one group has the larger value than a randomly drawn member of the other, ties counted as half: .455 that a Riverside member waits longer than a Station one, or .545 the other way. It is the common language effect size under its own name, ½ is no effect, and it is the quantity Brunner-Munzel tests – p̂ = ½ – which is not the same hypothesis as equal medians, and can be rejected between two groups whose medians agree.

Spread as the hypothesis #

Dispersion test is a test whose hypothesis is on the spread rather than the location: are the groups equally variable. The F-test for equality of variances, Bartlett’s test and the Fligner–Killeen test are defined on the assumptions page as checks, gatekeepers for the tests on means; the comparison module’s Independent samples – variance family offers the same three as tests in their own right, and the difference is the question, not the arithmetic. spend_before by branch gives a variance ratio of .54 [.36, .80], F(100, 98), p = .002 – Station’s variance is 1.86 times Riverside’s – which read as a check demotes Student’s t and steers to Welch’s, and read as a finding says Station’s members are the less predictable spenders, a fact about the branches worth reporting on its own. Consistency is the hypothesis whenever a process, a teaching method or a treatment is meant to reduce variability rather than move the mean; the F-test can be run one-sided for it, and normality matters more here than for the tests on means, since a variance follows a heavy tail far more steeply than a mean does.

One variable against a number #

Reference value – μ₀ – is the number a one-sample test compares a variable’s location with, and it comes from outside the data: a target, a norm, a contract, a published mean. The app’s Reference value (μ₀) field takes one for every selected variable, or one per variable. The mean wait against five minutes: 5.20 [4.77, 5.62], t(184) = 0.91, p = .363; the one-sample Wilcoxon’s pseudo-median 4.95 [4.55, 5.40], V = 8,294.5, p = .864; the sign test’s 87 of 183 waits above five, two ties at five dropped, p = .554, with the median of 4.9 [4.3, 5.3] at an achieved coverage of 96%. Three tests, three locations, one answer: the waits sit at five minutes. The guide’s catalogue entry says which of the three to run.

Goodness-of-fit test compares one categorical variable’s observed counts with the counts an expected distribution predicts, as a chi-square over the categories with one degree of freedom fewer than there are categories. Are the members split evenly across the branches – 101 and 99 against 100 and 100: χ²(1) = 0.02, p = .888, Cohen’s w = .01. Are the age bands in the 1 : 2 : 1 shape a census might predict – 47, 102 and 46 against 48.75, 97.5 and 48.75: χ²(2) = 0.43, p = .808, w = .05. Is the recommendation an even split – 135 and 65 against 100 and 100: χ²(1) = 24.5, p < .001, w = .35. It is the only one-sample test for a categorical variable, and it asks about the whole distribution; a question about a single rate is the hypothesised proportion’s. When a category’s expected count falls below 5 the p is simulated instead.

Expected distribution is the hypothesis a goodness-of-fit test states, entered in the app as one relative weight per category – 1, 2 and 1 for the age bands – which the editor normalises to probabilities, .25, .5 and .25, and the test multiplies by the sample size into expected counts. Weights, not probabilities: a fair die is six ones, a 9 : 3 : 3 : 1 cross is those four numbers, published proportions are typed as they are. A category the sample never contained can be added at an observed count of zero, so a model can be penalised for a level that should have appeared, and a weight of zero is refused, since a category expected never to occur makes the chi-square undefined.

Hypothesised proportion – π₀ – is the rate a single category’s share is tested against by the one-sample proportion test, an exact binomial test: is the share of members who would recommend the café above the 60% a franchise might require. 135 of 200 is .675 [.61, .74], p = .030, Cohen’s h = .16 – yes, by a small margin – and the interval is the one the exact test inverts, so it excludes π₀ exactly when the test rejects. It is entered once for every selected variable in the app’s Hypothesised proportion (π₀) field.

Success category is the category whose share the proportion test is about – yes, for the recommendation – chosen per variable in the app under Category counted as a success, every other category counting as a failure. On a binary variable the other choice gives the mirror answer, .325 against .4; on a variable with more categories it decides which marginal share is tested, which is why the card repeats it in a Category column beside the result. “Is the rate 40%?” is only a question once it says the rate of what.

Paired categorical outcomes #

The design is the repeated measures design with a categorical answer: the same members asked a yes/no or a multi-category question on two or more occasions, and the question whether the answers moved. The café asked once, so its instances are told in words – had the members been asked whether they would recommend the café before the menu change and again after.

McNemar’s test is the two-occasion test for a binary answer, built on the discordant pairs alone: the members who said yes before and no after, against those who moved the other way. The members who answered the same way twice carry no information about change, however many they are, so the test reads the split among the switchers – a lopsided split is a change, an even one is not – and it uses the exact binomial distribution below 25 discordant pairs, with the continuity correction above. A patient’s symptom present or absent at two visits, a voter’s intention in two polls, a screening test’s verdict before and after a training course: the pattern is one subject, one binary answer, two occasions. With three or more occasions it auto-expands to a pairwise table.

Stuart-Maxwell test is the same for an answer with three or more categories – a recommendation of definitely, maybe or no at two occasions – testing whether the distribution over the categories is the same at both, marginal homogeneity, from the cells off the diagonal of the occasion-by-occasion table. On a binary answer it reduces to McNemar’s, and the app hands it over.

Cochran’s Q test is the binary answer over three or more occasions – would recommend at three visits – the within-subjects counterpart of a chi-square test, built on the informative subjects: those whose answer varied across the occasions, since a member who said the same thing every time contributes nothing. Its post-hoc is a McNemar test per pair of occasions.

A nurse: ward → between-subjects factor, admission and discharge → within-subjects factor, patient → subject ID, and age → covariate; a teacher: teaching method and class year → the two factors of a crossing, prior grade → covariate, the usual method → control group; a product team: page version → factor, each visitor’s three sessions → repeated measures, visitor → subject ID; a clinician: a battery of symptom scores → the dependent variables of a MANOVA, dose taken at three levels → an ordered factor, study centre → stratifying variable, a symptom present or absent at two visits → a paired categorical outcome.

Where this appears in DataSuite #

• Comparison analysis – the Analysis type radios and the variable roles spell the design; the groups preview counts cells and complete subjects; the analysis strategy chooses between crossing the factors and combining them; Sums of squares and Within-subjects approach set the rules above; the post-hoc tests panel lists the families; the factorial, mixed, ANCOVA and heterogeneous slopes cards print the effects, the estimated and adjusted means and the regions; the within-subject interval is a summary statistic and an error bar. The families beyond the decision table are the independent-samples, variance, one-sample, categorical, paired categorical and mixed test lists; the MANOVA / MANCOVA card with its discriminant analysis section and the ART-ANOVA card; correspondence analysis with its tables and map; the stratifying variable role; Trend direction, Trim level, the expected-distribution editor and the hypothesised proportion panel; the Kolmogorov-Smirnov card’s where the distributions separate table and the ECDF plot.
• Correlation analysis – mutual information, Theil’s U and the entropies as measures of association between two variables, where the comparison module tests them between groups.
• Data transformation – the table converter that stacks a wide file into the long one a within-subjects factor needs, and the binning that made the age bands.
• Regression analysis – the same crossing as a model with dummy-coded factors and a product term, the ANOVA table under Type II, and moderators with their simple slopes and regions.
• Analysis planner – the sample a one-way or factorial design needs for a given Cohen’s f.
• Settings – the p-value adjustment the run-wide post-hoc families fall back on, and the confidence level every interval on this page is read at.