Comparison analysis #

The Comparison analysis module tests whether groups or conditions differ on one or more variables. Pick the design that matches how the data was collected – separate groups, the same subjects under several conditions, both at once, or one variable against a fixed number – assign the roles the design needs, choose a test, and read one card per test with its effect sizes, post-hoc comparisons, assumption checks and plots.

What does “comparing groups” mean statistically? Whether the difference seen in the sample is larger than chance alone would produce – the test answers with a p-value against the null hypothesis of no difference, and the guide walks from the question to the test.

1. Choose an analysis type (independent, dependent, mixed, or one sample)
2. Assign variable roles (grouping, conditions, subject ID)
3. Pick a statistical test
4. Configure options (effect sizes, post-hoc, summary stats)
5. Optionally check assumptions, then click Run comparison analysis

Analysis types #

Analysis type. Four radios, one per design. The choice decides which role lists the panel shows and which tests the test select offers – the only thing that hides a test.

• Independent samples – the groups are separate, unrelated cases, each in one group only: treatment vs. control, one site vs. another. Roles: one or more grouping variables – see between-subjects factor.
• Dependent samples (repeated measures) – the same cases measured under two or more conditions, one row per case per condition: pre-test vs. post-test, three time points. Roles: one or more condition variables and a subject ID – see repeated measures.
• Mixed model (between + within subjects) – both at once: separate groups, each measured under every condition – treatment vs. control, each at pre and post. Roles: grouping variables, condition variables and a subject ID – see mixed design.
• One sample (vs. reference value) – no groups: every selected variable is tested on its own against a fixed number – test scores vs. a 70-point pass mark, a measurement vs. its calibration target. No roles; the numeric tests take a reference value, the categorical ones an expected distribution or a hypothesised proportion – see one variable against a number.

Why does the design matter? A dependent design uses each case as its own control, so the differences between cases leave the test; an independent test on paired data throws that away, and a dependent test on unrelated cases pairs rows that have nothing in common – see within-subjects factor and how the cases are related.

Long format required: every design but one sample expects one observation per row, and the panel says so under the radios. If conditions sit in separate columns (wide format), the Convert to long format button opens the table converter – see wide and long.

Setting up #

Variable roles #

The variables chosen in the Variables dialog take their roles under Variable selection; the design decides which lists appear, and a click selects an item. A variable can fill only one role – the run refuses one placed in two.

• Grouping variables – the between-subjects factors of an independent or mixed design: the categorical variables whose levels are the groups – see between-subjects factor. Each needs at least two levels. With two or more selected, a native factorial test fits their crossing and every other test asks for an analysis strategy.
• Condition variables (within-subjects) – the within-subjects factors of a dependent or mixed design: the variables saying which condition or occasion a row belongs to – see within-subjects factor. Each needs at least two levels, and the rows of one subject are matched through the subject ID.
• Subject ID – the variable saying which rows belong to the same subject, so that its measurements are matched across conditions – see subject ID. Required for dependent and mixed designs, not offered for independent or one-sample ones. A selected column named like subject ID (Subject ID, subject_id, subj id – the table converter generates one) is selected automatically.
• Select a variable – the placeholder: a dependent or mixed design does not run until a subject ID is chosen.
• Reference value (μ₀) – one-sample designs, for the numeric tests (one-sample t, Wilcoxon signed-rank, sign): the fixed number every selected variable is tested against – see reference value. For the Chi-square goodness-of-fit test the field gives way to the expected-distribution editor, for the One-sample proportion test to the hypothesised-proportion panel; the three sub-panels are mutually exclusive, and the selected categorical variables are those two tests’ dependent variables.
• Set per variable – gives each selected variable its own μ₀: a list with one field per numeric variable appears, pre-filled with the shared value, and the shared field then only seeds newly added rows – its hint says which.
• Covariates – numeric variables held fixed while the groups are compared – see covariate and ANCOVA. The list offers numeric variables only and appears for the tests that take covariates: ANCOVA (Analysis of Covariance) and MANCOVA (Multivariate ANCOVA), which need at least one, and Factorial ANOVA, Mixed ANOVA, Mixed MANOVA, Repeated measures ANOVA, Repeated measures factorial ANOVA and Repeated measures MANOVA, which accept them. A repeated measures ANOVA given covariates is fitted on the repeated measures MANOVA machinery and reported as the univariate test you selected – one F per effect, the effect size and its interval, the Greenhouse-Geisser and Huynh-Feldt corrections, the covariate effects – with post-hoc condition contrasts on covariate-adjusted estimated marginal means.
• Stratifying variable – one categorical variable the Cochran-Mantel-Haenszel test pools its association across – see stratum. Shown for that test alone, which needs it with at least two levels; the chosen variable leaves the dependent-variable pool, as covariates and the subject ID do.
• None – the default: no stratifying variable, and the Cochran-Mantel-Haenszel test does not run until one is chosen.

What is a covariate? A variable you suspect influences the outcome but are not studying – prior GPA when comparing teaching methods – whose influence the model removes before the groups are compared; see adjusting for a covariate.

What is a stratifying variable? A grouping held constant while an association is tested – a study centre, an age band, sex – so that a real association is neither washed out nor manufactured by differences between the strata; see stratum and Simpson’s paradox.

Every selected variable in no role is a dependent variable – the outcome compared across the groups or conditions, as the note under the card title says – see dependent variable. A test that accepts only numeric (or only categorical) dependent variables drops the ones of the other type before the run and a warning names them, so a partial exclusion never passes unnoticed; a selection that leaves no dependent variable stops the run.

Groups preview #

Below the roles, a live preview describes the design the run will fit.

For independent and mixed designs it lists the groups with their sizes – a group under 3 in warning colour, under 10 muted, the first 30 shown – and, from three groups up, the number of pairwise comparisons the run will make, when the test is a two-sample one or post-hoc tests are on. Warnings follow, ten at most: a group with fewer than 3 observations, an empty cell, a numeric grouping variable with more than 50 distinct values (it is probably continuous, not categorical), a crossing of more than 1,000 cells, more than 100 pairwise comparisons, and a crossing over the 10,000-cell cap, which the run refuses. With two or more grouping variables the groups are the cells of the full crossing – the units a native factorial test or Combine variables into a single factor compares – with empty cells dropped and named; under Analyze each variable separately every grouping variable is listed on its own. In a mixed design a cell’s n is its distinct subjects, not its rows.

For dependent designs the preview appears once both a condition variable and a subject ID are selected. It shows, per condition variable, the number of conditions × the number of complete subjects – those with a row in every condition – then a line counting the subjects excluded for missing a condition and a line counting those with more than one row in a condition, whose rows the run averages into one value per cell – see complete subjects. Under Combine variables into a single factor the preview shows the crossed condition cells instead.

When two or more condition variables are not fully crossed – some combination of their levels has no rows – the preview names the missing combinations, and a test that crosses within-subjects factors (repeated measures factorial ANOVA, its ART form and the mixed family) runs the observed cells as one within-subjects factor, the within main effects and interaction no longer separable; the card says so again.

Multiple grouping or condition variables #

Analysis strategy. Two radios above the test select, shown when two or more grouping variables (independent design) or condition variables (dependent design) are selected and the chosen test takes a single factor; a native factorial test fits every factor at once and hides the choice, and a mixed design always fits its grouping and condition variables together. A preview under the radios counts the analyses or cells the run will make.

• Analyze each variable separately – one run of the chosen test per grouping variable (independent) or condition variable (dependent), each its own card, the p-values of every run adjusted as one family under the global adjustment setting. The only strategy the trend tests (Jonckheere-Terpstra, Page’s, Cochran-Armitage) offer.
• Combine variables into a single factor – crosses the selected factors into one factor whose levels are the cells (Gender × Treatment, Time × Dose) and runs the chosen test on it as a single-factor omnibus – no main effects, no interaction – see full crossing. Empty cells are dropped; a crossing of more than 10,000 cells, or one with no occupied cell, stops the run with a message.

Separate, combined, or native? Separate when the factors are independent questions, combined when the cells themselves are the groups to compare, native – factorial ANOVA, repeated measures factorial ANOVA, ART-ANOVA – when the question is each factor’s main effect and their interaction.

Choosing a test #

The Statistical test select lists every test of the selected analysis type, and that choice is the only thing that hides a test: every other precondition – an ordinal factor for a trend test, two or more grouping variables for factorial ANOVA, two or more dependent variables for the MANOVA family, a covariate for ANCOVA – leaves the test selectable, and a run that does not meet it stops at Run comparison analysis with a message naming what to change, so every test stays discoverable. The tables below compare the tests of each family, and each test’s entry says what it needs, what it offers and what its card reports; the guide walks from the question to the test.

Parametric or non-parametric? A parametric test assumes a shape for the data and buys power with it; a rank-based test assumes less and needs a larger sample for the same effect – see choosing between the families, and let Check assumptions make the call: its recommendations demote a test whose assumption fails and name the partner to run instead.

A sharper question, or a broader one? The trend tests ask whether the values rise or fall across ordered levels and spend their power on that alone – see trend test; the whole-distribution and information-theoretic tests ask whether the distributions differ in any way, at the cost of not saying how – see whole-distribution tests.

Two-sample tests on more than two groups. A two-sample test – the t-tests, Mann-Whitney, Brunner-Munzel, Yuen’s, Kolmogorov-Smirnov, the F-test, the paired two-condition tests, McNemar’s and Stuart-Maxwell – chosen for a factor with three or more levels runs every pair directly, with no omnibus first, and the pairs’ p-values enter the run-wide p-value adjustment. The independent samples t-test computes each pair against the within-group error pooled over all the groups; Welch’s keeps a separate-variance term per pair; every other test runs each pair on its own data. The pairs land in the pairwise comparisons card.

When a test refuses to run #

Preconditions are checked when you press Run comparison analysis, before anything is fitted, and the run stops with a message naming what to change – the module declining to return a number it cannot stand behind, not a failure of the data. The refusals you are most likely to meet:

• A trend test on a nominal factor, or on fewer than three levels. The Jonckheere-Terpstra, Page’s and Cochran-Armitage tests read their order from the factor’s ascending numeric values, so the factor must be of the ordinal measurement type – the message names the variable to recode, which data transformation does – and must have at least three levels; under two or more factors they run with Analyze each variable separately only, never on a combined factor.
• A role the test needs left empty. Two or more dependent variables for the MANOVA family, two or more grouping variables for factorial ANOVA, two or more condition variables for the repeated measures factorial tests, a covariate for ANCOVA and MANCOVA, a stratifying variable for Cochran-Mantel-Haenszel, a binary dependent variable for McNemar’s, Cochran’s Q and Cochran-Armitage – each message names the test and the role.
• A covariate, or a mixed design’s grouping variable, that varies within a subject. In a dependent or mixed design a covariate takes one value per subject, and in a mixed design so does every grouping variable; the message names the variable.
• Not enough subjects for a within-subjects multivariate fit. Repeated measures MANOVA, Mixed MANOVA and a repeated measures ANOVA given covariates stop when the residual degrees of freedom fall below the within-contrast dimension – the message gives both numbers and the three ways out: fewer conditions, fewer dependent variables or covariates, more subjects.
• A group too small for its own covariance matrix. Johansen’s heteroscedastic MANOVA needs more complete observations in every group than there are dependent variables, and dependent variables that stay linearly independent within every group; the message names the groups at fault and points to MANOVA, which pools one matrix.
• A permutation run that would not finish. The energy test refuses a sample beyond the work it can complete – past roughly 1,300 rows at two dependent variables – and names MANOVA, Johansen’s and the permutation replications setting as the ways forward.
• An empty between-subjects cell under Type III. A factorial or mixed design with two or more grouping variables and an empty cell in their crossing is refused under Type III sums of squares; under Type II it runs, with the interaction row blank and a warning.

A within-subjects crossing with a missing combination is not refused: the run collapses the observed cells into one within-subjects factor and says so – see the groups preview.

Independent samples – numeric #

• Independent samples t-test – two groups’ means, the groups assumed to share one spread (Student’s t) – see the guide. Offers the one-tailed and equivalence directions and the ROC block; effect sizes Cohen’s d, Hedges’ g, Glass’ Δ, the point-biserial r, η² and ω². With three or more groups every pair’s t is computed against the within-group error pooled over all the groups.
• Welch’s t-test (unequal variances) – two groups’ means without the equal-spreads assumption, the safer default and the test the assumption check recommends when Levene’s fails – see Welch’s correction. The same directions and ROC block; effect sizes Welch’s d and g, Glass’ Δ and the point-biserial r – the pooled-SD d is not offered. With three or more groups every pair keeps its own separate-variance error term.
• Mann-Whitney U test – two groups on ranks: does one group tend to have the higher values – see the guide. Its location-shift reading assumes the two groups share a shape, and the check steers to Brunner-Munzel when their spreads differ. One-tailed and equivalence directions, the continuity correction and the ROC block – the AUC is U rescaled; effect sizes the rank-biserial r, Cliff’s δ and the common-language effect size, with the Hodges-Lehmann shift and its interval on the card.
• Brunner-Munzel test – two groups on ranks with no shared-shape assumption: it tests the relative effect p̂ against ½ and stays valid when the spreads or shapes differ – see Brunner-Munzel. At small samples the p comes from the exact permutation distribution, the statistic and df columns stay blank and a note says so. One-tailed directions and the ROC block; effect sizes the common-language effect size (p̂ itself) and Cliff’s δ.
• One-way ANOVA – three or more groups’ means at once on the F ratio, equal spreads assumed – see the guide. Post-hoc pairs by the exact multivariate-t, Tukey HSD, pooled t-tests or Dunnett’s test against a control; effect sizes η², ω² and ε².
• Welch’s one-way ANOVA – the k-group omnibus without the equal-spreads assumption, the parametric test the check keeps recommended when Levene’s fails on three or more groups – see the guide. Post-hoc Games-Howell or Dunnett’s T3; its effect size is the explanatory measure ξ – η², ω² and ε² are not offered.
• Brown-Forsythe one-way ANOVA (F*) – the other heteroscedastic omnibus: Welch’s question with the group variances weighted by $1-n_i/N$ rather than by precision, which can behave better when the groups are skewed; the two usually agree, and reporting both is a sensitivity check. Post-hoc Games-Howell or Dunnett’s T3; effect size ξ. Not the Brown-Forsythe of checking assumptions, which is the median-centred Levene’s test of equal spreads – this test compares the means.
• Yuen’s trimmed-means t-test – two groups’ trimmed means, 20% off each tail by default under Trim level, robust to heavy tails and unequal spreads at once – see Yuen’s test. Two-sided only; the ROC block; effect sizes ξ and the robust standardized difference δ_t. With three or more groups every pair runs on its own data.
• Trimmed-means one-way ANOVA (t1way) – the k-group trimmed-means omnibus, the robust counterpart of Welch’s ANOVA at the same trim level – see t1way. Post-hoc the pairwise trimmed-means comparisons (lincon); effect size ξ.
• Kruskal-Wallis test – three or more groups on ranks, the rank-based partner of one-way ANOVA – see the guide. Post-hoc Dunn’s test, all pairs or against a control; effect sizes η²_H and ε².
• Kolmogorov-Smirnov test – two groups’ whole distributions: D is the largest gap between their ECDFs, so it reacts to a shift, a spread or a shape alike – see Kolmogorov-Smirnov D. Two-sided only, and D is its own effect size; the card adds the where the distributions separate table (D⁺, D⁻ and the value each is reached at) and marks the gap on the ECDF plot. The p is exact until n₁·n₂ reaches 10,000 and asymptotic beyond it, with a note. With three or more groups every pair runs on its own data.
• Anderson-Darling k-sample test – the same question for two or more groups at once, every gap counted and the tails weighted – see Anderson-Darling. Reports the standardized statistic T.AD, its own effect size; with ties, or a group under 5 observations, the p is simulated over the permutation replications and a note records the count, and a pooled sample too small to rank reports an error. Post-hoc: a two-sample Anderson-Darling test per pair.
• Jonckheere-Terpstra test – a monotone trend in a numeric outcome across three or more ordered groups, sharper than Kruskal-Wallis when the order is real – see trend test. Needs an ordinal grouping factor and takes its direction from Trend direction; effect size Kendall’s τ-b. The p is by permutation when the values carry ties or n ≤ 100, with the count noted.
• Mutual information test – how much knowing the group tells you about the outcome, in bits, tested by permutation – see information-theoretic tests. An omnibus over two or more groups that reacts to any change in the distribution; the p follows the permutation replications setting, with the count noted; effect size Theil’s U, and a legend under the statistic states its ceiling. Post-hoc: the same estimate per pair. Independent designs only.
• Jensen-Shannon divergence test – the distance between the groups’ distributions and their mixture, in bits, tested by permutation – see Jensen-Shannon divergence. An omnibus bounded by log₂ k for k groups, which the legend states; effect size the normalized divergence; post-hoc the pairwise Jensen-Shannon divergence tests. Independent designs only.
• Factorial ANOVA – two or more grouping variables in one model: a main effect per factor and their interaction – see the guide. Accepts covariates (the card then says factorial ANCOVA), takes Sums of squares and per-effect post-hoc contrasts; effect sizes partial η², η², ω² and ε². Its card is read under factorial ANOVA.
• ANCOVA (Analysis of Covariance) – groups compared at a common value of one or more covariates – see ANCOVA. Needs at least one covariate; takes Sums of squares and covariate-adjusted post-hoc contrasts; effect sizes partial η², η², ω² and ε². Its card – the covariate rows, the adjusted means, the slopes – is read under ANCOVA.
• MANOVA – several dependent variables compared together, as a profile – see MANOVA. Needs two or more dependent variables; the card reports the four multivariate statistics, the univariate follow-ups and, with Include descriptive discriminant analysis, the discriminant functions; post-hoc per-variable contrasts by the exact multivariate-t, Tukey HSD or pooled t-tests; effect sizes η², ω² and ε². Read under MANOVA / MANCOVA.
• MANCOVA (Multivariate ANCOVA) – MANOVA with one or more covariates held fixed – see MANCOVA. Needs two or more dependent variables and at least one covariate; the same follow-ups, discriminant analysis and post-hoc; effect sizes partial η², η², ω² and ε².
• Johansen’s heteroscedastic MANOVA – the multivariate Welch’s test: each group weighted by its own covariance matrix, the test the check steers to when Box’s M fails – see Johansen’s. Needs two or more dependent variables and more complete observations per group than dependent variables; the omnibus carries no effect size, its follow-ups are Welch’s ANOVAs with ξ, and its post-hoc is Games-Howell or Dunnett’s T3 on them.
• Energy test of equal distributions – do the groups share one joint distribution at all, by permutation and with no distribution assumed – the fallback the check steers to when multivariate normality fails, and not a MANOVA substitute, since a difference in spread alone rejects – see energy test. Needs two or more dependent variables, which are standardized first; the replicate count trades down against a fixed work budget before the run is refused, never below 199, and the card discloses the count used. No post-hoc; the follow-ups are per-variable energy tests, and the effect size is R²ₑ, the share of the distances that lies between the groups.

Independent samples – variance #

These tests ask a different question from every test above: not whether the groups differ in location but whether they differ in spread – see dispersion test.

• F-test for equality of variances – two groups’ variances as a ratio, s₁²/s₂², with a confidence interval – see F-test. Assumes normality, and the check demotes it when normality fails. Can be run one-sided under Test direction, where Group 1 > Group 2 is read on the variances, not the means; effect size the variance ratio. With three or more groups every pair runs on its own data.
• Bartlett’s test – the k-group omnibus on the variances, assuming normality in every group and sensitive to its failure at any sample size – see Bartlett’s test. Two-sided only, no effect size; post-hoc: a two-group Bartlett’s test per pair.
• Fligner-Killeen test – the k-group omnibus on ranks, robust to non-normality and the test the check keeps recommended when normality fails – see Fligner-Killeen. Two-sided only, no effect size; post-hoc: a two-group Fligner-Killeen test per pair.

Not Levene’s test. Levene’s test (median-centred) runs under checking assumptions as the gatekeeper for the tests on means; the tests here make unequal spread the hypothesis – see spread as the hypothesis.

Dependent samples – numeric #

• Paired samples t-test – two conditions on the same subjects, tested on the mean of the paired differences – see the guide. One-tailed and equivalence directions; effect sizes Cohen’s d_z and Hedges’ g_z on the differences’ SD, d_av and g_av on the conditions’ SDs. With three or more conditions every pair runs on its own data.
• Wilcoxon signed-rank test – two conditions on the ranked magnitudes of the differences, for differences that are skewed or carry outliers; assumes they are symmetric around their centre, and the symmetry check steers to the sign test when they are not – see the guide. One-tailed and equivalence directions, the continuity correction; effect sizes the matched-pairs rank-biserial r and Wilcoxon r. With three or more conditions every pair runs on its own data.
• Paired sign test – counts how many pairs moved up and how many down and nothing else, so no shape assumption survives – see the guide. Pairs with a zero difference are dropped as ties and counted under the table; the card reports the positive count out of the effective n, the median of the differences with a distribution-free interval whose achieved coverage is noted – at very small n it is the whole line – and r₊ = 2p̂ − 1 as the effect size, 0 for an even split and ±1 when every pair moves the same way. One-tailed directions only.
• Repeated measures ANOVA – three or more conditions on the same subjects, the within-subjects one-way ANOVA, with sphericity as its assumption and the Greenhouse-Geisser and Huynh-Feldt corrections on the card – see the guide. Takes covariates, which reroute the fit as Variable roles says, and the Within-subjects approach; post-hoc condition pairs by the exact multivariate-t, paired t-tests or Dunnett’s test; effect sizes partial η², generalized η² and partial ω².
• Friedman test – three or more conditions on ranks within each subject, the rank-based partner of repeated measures ANOVA – see the guide. Post-hoc pairwise Wilcoxon signed-rank, Conover’s or Nemenyi tests; effect size Kendall’s W.
• Repeated measures factorial ANOVA – two or more condition variables crossed within subjects: an F per within main effect and interaction, each with its own sphericity correction – see full crossing. Needs two or more condition variables; takes covariates (the card then says factorial ANCOVA), Sums of squares and the Within-subjects approach; per-effect post-hoc contrasts; effect sizes partial η², η², ω² and ε². A crossing with a missing combination is collapsed into one factor, as the groups preview warns.
• Repeated measures factorial ART-ANOVA – the same crossing on aligned ranks, for residuals far from normal: a rank-based F per within main effect and interaction. Needs two or more condition variables, takes no covariates, and its post-hoc method chooses only the correction; effect size partial η². Read its card under ART-ANOVA.
• Page’s trend test – a monotone trend across three or more ordered conditions, the within-subjects counterpart of Jonckheere-Terpstra – see trend test. Needs an ordinal condition factor and a direction from Trend direction – it has no two-sided form; effect size the mean within-block ρ against the hypothesised order.
• Repeated measures MANOVA – several dependent variables across conditions, modelled jointly, with a univariate follow-up per variable – see MANOVA. Needs two or more dependent variables and enough subjects for the within-contrast dimension, as refusals says; takes covariates; post-hoc per-variable condition contrasts by the exact multivariate-t, paired t-tests or Dunnett’s test; effect sizes partial η², η², ω² and ε².

One sample – numeric #

Each test compares one variable’s location with the reference value μ₀ – a fixed number from outside the data – and reports a confidence interval for the location itself, the mean, the pseudo-median or the median, with the μ₀ each variable was tested against as a column, which matters when Set per variable gives each its own. The assumption check tests each variable’s normality and steers to the t-test or to the rank-based tests accordingly.

• One-sample t-test – the mean against μ₀, the most powerful of the three when the data is roughly normal – see the guide. One-tailed and equivalence directions, the standardized bound in the variable’s own SD; effect sizes Cohen’s d and Hedges’ g as $(\text{mean}-{\mu }_0)/\text{SD}$.
• One-sample Wilcoxon signed-rank test – the pseudo-median against μ₀, on the ranked magnitudes of the deviations from it. One-tailed and equivalence directions, the standardized bound in the variable’s MAD, the continuity correction; effect sizes the matched-pairs rank-biserial r and Wilcoxon r.
• Sign test – the median against μ₀, counting only how many observations fall above it, so it holds for ordinal data and heavy outliers at a cost in power. Ties at μ₀ are dropped and counted under the table; the card reports the count above μ₀ out of the effective n, the distribution-free interval for the median with its achieved coverage noted – the whole line at very small n – and r₊ = 2p̂ − 1 as the effect size. One-tailed directions only.

One sample – categorical #

The only one-sample tests for a categorical variable; selecting either swaps the Reference value (μ₀) field for its own panel and restricts the dependent variables to categorical ones (an ordinal item qualifies). An association between two categorical variables is the chi-square test of independence’s question, not theirs.

• Chi-square goodness-of-fit test – how often each category occurred against how often the expected distribution says it should, over three or more categories or two: a fair die, a 9:3:3:1 cross, a 40/35/25 split – see goodness-of-fit test. The editor takes one relative weight per category, normalised to probabilities. Two-sided; effect size Cohen’s w; when the expected counts are too thin for the approximation the p is simulated, and a note says so.
• One-sample proportion test – one category’s share against π₀, an exact binomial test: is the response rate really 40%, is the defect share above the 2% the contract allows – see hypothesised proportion. You enter π₀ and pick the category counted as a success per variable, every other category counting as a failure – on a variable with more than two categories it is that level’s marginal share, printed in a Category column. The interval beside p̂ is the Clopper-Pearson interval, built on the same binomial probabilities as the test, so the two read the same evidence and nearly always agree; one-tailed directions; effect size Cohen’s h, on Cohen’s 0.2 / 0.5 / 0.8 scale.

Goodness-of-fit or proportion test? Goodness-of-fit when the hypothesis is the whole distribution, the proportion test when it is a single rate and you want an exact p and an interval on p̂ – on a binary variable the two overlap and the proportion test is the sharper tool.

Independent samples – categorical #

• Chi-square test of independence – whether the dependent variable’s distribution differs between the groups, from the table of counts against the counts expected under independence – see the guide. The continuity correction on a 2 × 2 table, Cochran’s rule on the expected counts noted when the table is thin; effect sizes Cramér’s V and, on a 2 × 2 table, the odds ratio; post-hoc: a chi-square test per pair of groups; Include correspondence analysis decomposes the table.
• Fisher’s exact test – the same question with a p computed from the exact distribution of the table, for expected counts too small for chi-square – see the guide. No test statistic; exact on a 2 × 2 table at any size and on larger tables up to 30 cells (1,000 rows once both sides exceed two categories), beyond which the p is simulated over 10,000 replicates with a note. One-tailed directions on a 2 × 2 table only; effect sizes Cramér’s V and, on a 2 × 2 table, the odds ratio with its exact interval; post-hoc per pair, and correspondence analysis as for chi-square.
• Cochran-Armitage trend test – a linear trend in a binary outcome’s rate along three or more ordered groups, sharper than a 2 × k chi-square when the order is real – see trend test. Needs an ordinal grouping factor and a binary dependent variable; the trend scores are the factor’s own numeric values, so unequal spacing is honoured, and the modelled event is the higher of the two outcome values – a note under the table records both. The direction is Trend direction’s; a signed Z is the statistic; effect size Cohen’s w.
• Cochran-Mantel-Haenszel test – the association between the dependent variable and the grouping variable within the levels of a stratifying variable, pooled across them – see Cochran-Mantel-Haenszel. Needs exactly one grouping variable and a stratifying variable with two or more levels; reports a χ² with (I − 1)(J − 1) df for any table shape, and on a 2 × 2 × K table the common odds ratio as its effect size, the Breslow-Day check and the continuity correction; a stratum with one observation is left out, and a 2 × 2 × K table too sparse for the approximation switches to the exact conditional test, with a note.

Dependent samples – categorical #

• McNemar’s test – a binary outcome at two conditions, tested on the discordant pairs alone – see McNemar’s test. Needs a binary dependent variable; below 25 discordant pairs it switches to the exact binomial test, above it the χ² with the continuity correction; effect sizes φ, the odds ratio and Cohen’s g. With three or more conditions every pair runs on its own data.
• Stuart-Maxwell test – marginal homogeneity for an outcome with three or more categories at two conditions, from the cells off the diagonal – see Stuart-Maxwell. On a binary outcome it runs McNemar’s test and the card says so; effect size Cohen’s g. With three or more conditions every pair runs on its own data.
• Cochran’s Q test – a binary outcome across three or more conditions, built on the informative subjects – see Cochran’s Q. Needs a binary dependent variable; post-hoc a McNemar test per pair of conditions; effect sizes Cohen’s w (Q) and Cochran’s average φ².

Mixed model #

All three take any number of grouping variables and any number of condition variables and report every main effect and every interaction of the full crossing – see mixed design; each grouping variable must take one value per subject.

• Mixed ANOVA – a between-subjects and a within-subjects factor, or several of each, in one model, each dependent variable on its own – see the guide. Takes covariates, Sums of squares and the Within-subjects approach; per-effect post-hoc contrasts by the exact multivariate-t, Tukey HSD, pooled t-tests or Dunnett’s test; effect sizes partial η², η², ω² and ε². Its card is read under mixed ANOVA.
• ART-ANOVA (Aligned Rank Transform) – the mixed design on aligned ranks, the non-parametric fallback when the residuals are far from normal, keeping the main effects and interactions separable where a plain rank transform would not. Strictly factorial: no covariates (a selected covariate is ignored and the card says so), no simple effects, a post-hoc method that chooses only the correction, and partial η² as its one effect size. Its card is read under ART-ANOVA.
• Mixed MANOVA – the mixed design with several dependent variables modelled jointly, a univariate follow-up per variable and adjusted means per effect – see MANOVA. Needs two or more dependent variables and enough subjects for the largest within-contrast dimension, as refusals says; takes covariates and Sums of squares; post-hoc per-variable, per-effect contrasts by the exact multivariate-t, pooled t-tests or Dunnett’s test; effect sizes partial η², η², ω² and ε². Read under MANOVA / MANCOVA.

Configuration #

Test direction #

• Test direction – the alternative hypothesis the p-value counts, for the tests that compare two things: the t-tests, Mann-Whitney, Brunner-Munzel, the paired and one-sample Wilcoxon signed-rank tests, the sign tests, the one-sample proportion test, the F-test for equality of variances and Fisher’s exact test on a 2×2 table. It is hidden for every omnibus, for Yuen’s test, which is two-sided only, for Kolmogorov-Smirnov, whose card reports D⁺ and D⁻ in place of a one-tailed p, and for Fisher’s exact test once the grouping variable or a selected dependent variable has more than two categories – Run comparison analysis re-checks the realized table and stops with a message when it is not 2×2. The Standard group holds the two-tailed and the two one-tailed directions; the Equivalence group is offered by the t-tests, Mann-Whitney and the two Wilcoxon signed-rank tests alone. The option labels name what the test orders: the groups in an independent design, the conditions in a paired one – by name, once a single two-level condition variable is selected – the variable against μ₀ (p̂ against π₀ for the proportion test) for one sample, and the variances for the F-test. In a pairwise expansion every pair is tested in the chosen direction, group 1 being the first of the pair’s two levels in the app’s level order.
• Two-tailed (Group 1 ≠ Group 2) – the default: a difference in either direction counts, and the p-value sums both tails – see two-tailed.
• One-tailed (Group 1 > Group 2) – the alternative that group 1 exceeds group 2: a larger mean or location, a larger variance under the F-test, greater odds of the first outcome category under Fisher’s exact test, where a note under the results spells the tested alternative out. Only that tail counts – see one-tailed.
• One-tailed (Group 1 < Group 2) – the mirror alternative, group 1 below group 2.

When is a one-tailed test legitimate? Only when the direction was fixed before the data were seen, and “no evidence” would be an acceptable answer had they run the other way – see one-tailed. When in doubt, keep two-tailed.

• Equivalence (TOST) – the alternative that the difference lies within ±Δ: two one-sided tests, one per bound, the larger of their p-values reported as p (equiv), and the difference’s interval built at 1 − 2α – see TOST and the (1 − 2α) interval.
• Non-inferiority – the alternative that group 1 is not worse than group 2 by more than Δ: one one-sided test against one bound, its side set by Preferred direction – see non-inferiority.
• Superiority – the alternative that group 1 beats group 2 by at least Δ: one one-sided test against the far bound, its side set by Preferred direction – see superiority.
• Minimal effect (MET) – the alternative that the difference exceeds Δ in either direction: two one-sided tests again, the smaller p reported as p (MET) – see minimal-effect test.
• Equivalence bound (Δ) – the margin every equivalence direction tests against, shown once one is selected: 0.5 by default, and the run stops with a message when it is blank or not positive. It is chosen before the data are seen – see equivalence margin – and its units are the radio’s beneath it.
• Raw – Δ in the dependent variable’s own units.
• Standardized (Cohen’s d) – the default: Δ in units of the spread the chosen test standardizes by, converted to raw units from the sample before the test runs, and the radio names the standardizer – Standardized (Cohen’s d) for the independent samples t-test and the one-sample t-test, Standardized (Welch’s d) for Welch’s t-test, Standardized (Cohen’s dz) for the paired t-test, and Standardized (robust spread) for the rank tests, which standardize by a median absolute deviation rather than a standard deviation. The card names the spread it used.
• Preferred direction – which tail carries the benefit, asked for non-inferiority and superiority alone; TOST and MET are symmetric in Δ. The bounds note under the results names the hypothesis that was tested – Non-inferiority (higher is better), Superiority (lower is better) – so the choice stays visible in the output.
• Higher values are better – the default: non-inferiority tests Group 1 − Group 2 > −Δ and superiority Group 1 − Group 2 > Δ.
• Lower values are better – for an error rate, a symptom score, a completion time: non-inferiority tests Group 1 − Group 2 < Δ and superiority Group 1 − Group 2 < −Δ.

Equivalent, or merely not different? A non-significant test never shows that two groups are the same; an equivalence direction turns the null around so that a significant result is positive evidence that any difference is smaller than Δ – see equivalence test and negligible.

Trend direction #

• Trend direction – replaces Test direction for the ordered-alternative tests – Jonckheere-Terpstra, Page’s trend test and Cochran-Armitage – and states the hypothesised direction of the trend across the ordinal factor’s ascending values; pick it from the hypothesis before the run, as for any one-tailed test – see monotone trend. Page’s trend test offers increasing and decreasing alone.
• Increasing trend – values rise as the level rises.
• Decreasing trend – values fall as the level rises.
• Any monotonic trend (two-sided) – a rise or a fall; not offered for Page’s trend test, and a two-sided direction reaching it from a restored configuration is refused at the run.

Expected distribution #

Shown for the Chi-square goodness-of-fit test in place of the reference value: one block per selected categorical variable, seeded with the variable’s observed categories at a weight of 1 each.

• Expected distribution – one relative weight per category – a proportion, a count or a ratio, in whatever form is easiest, since the weights are normalised before the test; the running Σ beside the variable’s name totals them and each row’s ≙ % shows the share its weight comes to. Every weight must be a positive number and the block needs at least two categories – a weight of 0, a blank or a negative stops the run with a message, as does a category listed twice or an added category left unnamed. See expected distribution for what the weights state.
• Add category – appends a category the sample never contained, with an editable name and weight; it enters the test at an observed count of 0.
• Remove category (✕) – drops an added category; an observed category cannot be removed.
• Reset to uniform – sets every weight back to 1, the “is it uniform?” null.

Hypothesised proportion #

Shown for the One-sample proportion test in place of the reference value:

• Hypothesised proportion (π₀) – the proportion every selected variable is tested against, one value between 0 and 1 shared by all of them, 0.5 by default; the run stops with a message when it is blank or out of range – see hypothesised proportion.
• Category counted as a success – one select per selected variable, listing that variable’s own categories; every other category counts as a failure, and each variable needs one chosen before the run. The card repeats the choice in a Category column – see success category for why the choice decides the question on a variable with more than two categories.

Post-hoc tests #

• Include post-hoc tests – adds the pairwise comparisons that say which levels differ once an omnibus has said that some do – see post-hoc test. Offered by the multi-group and multi-condition tests – one-way ANOVA, Welch’s and Brown-Forsythe ANOVA, the trimmed-means ANOVA, Kruskal-Wallis, the Anderson-Darling k-sample test, Bartlett’s test, Fligner-Killeen, the mutual information and Jensen-Shannon divergence tests, repeated measures ANOVA, Friedman, the factorial, repeated measures factorial and mixed ANOVAs, ANCOVA, ART-ANOVA and repeated measures factorial ART, the chi-square test of independence, Fisher’s exact test, Cochran’s Q, MANOVA, MANCOVA, repeated measures MANOVA, mixed MANOVA and Johansen’s heteroscedastic MANOVA – the checkbox appears once the factor has three or more levels, and for every factorial or mixed design, whose contrasts come per effect. The energy test has no post-hoc block and emits its per-variable follow-ups instead; Johansen’s compares on its per-variable Welch follow-ups, MANOVA and MANCOVA on each dependent variable’s univariate follow-up – covariate-adjusted under MANCOVA – and repeated measures and mixed MANOVA run condition contrasts per dependent variable, mixed MANOVA’s per effect across the between × within crossing. The results land in the pairwise comparisons card or in the per-effect tables of the factorial, ANCOVA, mixed, ART and MANOVA cards.
• Post-hoc method – the select beneath the checkbox, listing the methods the chosen test’s premises allow; the first listed is the default, so the exact multivariate t leads wherever it is offered. What each method corrects, and when it is the one, is the post-hoc comparisons section’s; the entries below say where each is offered and what it reports. A method that carries its own multiplicity control takes no further adjustment; the others take the p-value adjustment configured in the settings. The chi-square test, Fisher’s exact test, Anderson-Darling, Bartlett’s, Fligner-Killeen and the mutual information test show no select: their pairs run the omnibus test on each pair’s own sub-table or sub-sample, adjusted by the configured setting. For ART-ANOVA and repeated measures factorial ART the checkbox runs ARTool’s art.con contrasts either way and the select picks the correction alone – see ART-ANOVA results for the contrast types.
• Pairwise contrasts with exact multivariate-t adjustment – the default of one-way, factorial, repeated measures, repeated measures factorial and mixed ANOVA, ANCOVA, MANOVA, MANCOVA, repeated measures MANOVA and mixed MANOVA: Tukey’s pairs, calibrated on the exact multivariate t over the contrasts’ own correlation – see the concept. The correction is the method, so no separate adjustment is applied and the card’s Method line records “multivariate t (exact)”; each pair reports its difference, simultaneous interval, t and p.
• Tukey HSD – for the same ANOVA family, MANOVA, MANCOVA and the two ART tests: the studentized-range comparison of all pairs, controlling the family-wise error internally – see Tukey HSD. Pick it for parity with another package’s output. Interaction cell-pairs, for which Tukey is not defined, are corrected by the exact multivariate t instead.
• Pairwise t-tests using pooled ANOVA error – for the same tests: every pair on the omnibus’s pooled error term – difference, interval and t – with the configured p-value adjustment applied across the pairs, interaction cell-pairs included. When an adjustment is active the intervals widen to the Bonferroni simultaneous level, so under Holm, Hommel, Hochberg, Benjamini-Hochberg or Benjamini-Yekutieli a row can be significant while its interval still includes zero – a caveat under the table says so, and stays silent for a single-contrast family – see pairwise t-tests.
• Dunnett’s test (vs. control) – for one-way, factorial, repeated measures, repeated measures factorial and mixed ANOVA, ANCOVA, repeated measures MANOVA and mixed MANOVA, within-subject factors included: every level against the Control group, the k − 1 contrasts referred to the exact multivariate t and controlled internally – see Dunnett’s test. Reports the difference, interval, t and p per contrast, always in the long format.
• Games-Howell test – for Welch’s and Brown-Forsythe ANOVA and Johansen’s MANOVA: pairwise separate-variance t-tests on a studentized-range reference, valid under unequal variances and controlled internally – see Games-Howell.
• Dunnett’s T3 (vs. control) – for the same three tests: the many-to-one counterpart of Games-Howell, each contrast against the control on its own separate-variance t and Welch-Satterthwaite df, referred to the studentized maximum modulus and controlled internally – see Dunnett’s T3. Reports the difference, interval, t and p per contrast, in the long format.
• Pairwise trimmed-means comparisons (lincon) – for the trimmed-means ANOVA (t1way): pairwise trimmed-mean differences at the run’s trim level; the intervals control the family-wise error internally, the p-values take the configured adjustment – see lincon.
• Dunn’s test – for Kruskal-Wallis: all pairwise rank comparisons, with the configured adjustment – see Dunn’s test.
• Dunn’s test (vs. control) – for Kruskal-Wallis: Dunn’s z restricted to the contrasts against the Control group, the configured adjustment applied over those k − 1 alone, in the long format – see the concept.
• Pairwise paired t-tests – for repeated measures ANOVA: each pair of conditions a paired t-test on the subjects’ differences, with the configured adjustment – see the concept.
• Pairwise Wilcoxon signed-rank tests – for Friedman: each pair of conditions a Wilcoxon signed-rank test, with the configured adjustment – see the concept.
• Conover’s test – for Friedman: t-distributed pairwise comparisons on the rank sums, with the configured adjustment – see Conover’s test.
• Nemenyi test – for Friedman: the rank-sum differences on a studentized-range reference, controlled internally – see Nemenyi test.
• Pairwise McNemar tests – the sole method for Cochran’s Q: every pair of conditions compared with McNemar’s test – the exact binomial form below 25 discordant pairs, the uncorrected asymptotic χ² otherwise – with the configured adjustment across the pairs. Only p-values fill the grid, since McNemar’s statistic carries no direction, and each pair’s 2×2 table of agreements and discordances is shown beneath it – see the concept.
• Pairwise Jensen-Shannon divergence tests – the sole method for the Jensen-Shannon divergence test: the omnibus’s balanced-JSD estimator and permutation p-value on each group pair, with the configured adjustment across the pairs. A divergence is symmetric, so the matrix cells are not sign-flipped across the diagonal and no difference column appears; each pair draws its own replicates, so the run grows with the number of pairs and the permutation replications setting – see the concept.
• Control group – appears for the many-to-one methods – Dunnett’s test, Dunnett’s T3 and Dunn’s test vs. control – with one select per factor the run will fit: every grouping variable in batch mode, every factor of a factorial or mixed design, the within-subject factors too. Each lists that factor’s levels and defaults to the first; the other levels are compared against the one chosen, never against each other – see control group. When the chosen level is not a level of the variable being fitted, or has no usable data left once missing values are removed, the first level with usable data stands in and a line above the post-hoc table says so.

Many-to-one or all pairwise? A vs.-control method spends the error budget on the k − 1 contrasts against a genuine baseline – a placebo, a standard treatment – and keeps more power for them; the all-pairwise methods answer which of every pair differ – see all-pairwise and many-to-one comparisons.

Pairwise comparison format #

Shown as Pairwise comparison table format when the run produces a flat table of pairs – a two-sample test’s automatic expansion over three or more levels, or a post-hoc on a single-factor omnibus; the factorial and mixed designs lay their contrasts out per effect and never read it.

• Matrix format – the default: a matrix with the levels on both axes, each cell holding the pair’s statistic, p-value and, where enabled, effect size and interval – see matrix format under Reading results.
• Long format – one row per pair, with the group statistics, the difference and its interval, the statistic, p and effect size in columns – see long format.

The many-to-one methods – Dunnett’s test, Dunnett’s T3 and Dunn’s test vs. control – always render in the long format, whatever is chosen here.

Continuity correction #

• Apply continuity correction – shown for the chi-square test of independence, McNemar’s test, the Mann-Whitney U test, the Wilcoxon signed-rank test, the one-sample Wilcoxon signed-rank test, the Stuart-Maxwell test and the Cochran-Mantel-Haenszel test, and checked by default, so the p-values match R’s and SPSS’s defaults – see continuity correction for what it does and when to turn it off. For the table-based tests the correction exists on a 2×2 table alone, so the checkbox shows only while some table of the run can be one – a binary dependent variable beside a binary grouping variable, a two-cell crossing, or a factor with more levels that post-hoc tests will slice into pairs; the rank tests’ checkbox always shows. What it changes: for chi-square the Yates correction on a 2×2 table; for Cochran-Mantel-Haenszel the 2×2×K case alone; for McNemar’s the asymptotic χ² at 25 or more discordant pairs, below which the exact binomial test runs and the box has no effect; for Stuart-Maxwell the 2×2 case, where the test reduces to McNemar’s; for Mann-Whitney and the two Wilcoxon signed-rank tests the ±0.5 correction on the normal approximation, which takes effect only when the exact test is not used – ties, zero differences or a large sample. When a run’s realized table turns out not to be 2×2, or a rank test or CMH took its exact branch, a note under that variable’s results says the correction had no effect, so the checked box is not mistaken for a corrected p-value.

Trim level #

• Trim level (per tail) – shown for Yuen’s trimmed-means t-test and t1way: the percentage of each tail set aside before the means are compared, 0 to 49, at 20% by default – the smallest and largest fifth of each group’s values are left out and the middle 60% averaged – see trimmed mean. Less trimming keeps the estimate closer to the ordinary mean; more approaches a median-like comparison, at a cost in power on clean data; the run refuses a blank value or one of 50% or more. The one level applies to every group, to every pairwise Yuen comparison, to the lincon post-hoc and to the run’s ξ effect size.

Effect sizes #

• Include effect sizes – checked by default: adds an effect size beside every test statistic – how big the difference is, on a scale that does not grow with the sample – see effect size. The block is hidden for the tests that offer no measure, their statistic being a bounded magnitude in its own right – Kolmogorov-Smirnov, Anderson-Darling, Bartlett’s test and Fligner-Killeen – and for the energy test, whose card reports R²ₑ on the omnibus whatever the box says. For the multivariate tests the chosen measure goes to their univariate follow-ups, while the multivariate omnibus always reports the partial η² recovered from Pillai’s approximate F; Johansen’s heteroscedastic MANOVA estimates no pooled error term, so its omnibus carries no measure and the ξ you pick lands on its follow-ups alone. Assigning covariates to a repeated measures ANOVA reroutes it to a covariate-adjusted repeated measures MANOVA, where partial ω² has no counterpart – partial η² is computed and labelled instead, and a warning says so – and generalized η² is reported as the classical η².
• Effect size method – the select beneath the checkbox, listing the measures the chosen test offers – the table below, whose first-named measure is the test’s default; what each measures, its scale and its benchmarks are the effect sizes page’s, and the Interpretation column labels the magnitude against them. Under the multivariate within-subjects route the list narrows to Partial eta-squared, the one measure that route can compute.

Two measures print more than one number: Glass’ Δ is reported as Δ₁, standardized by the first group’s SD, and Δ₂, by the second’s, each with its own interval and magnitude label, since the module has no control-group selector to say which SD is the standardizer; and the Stuart-Maxwell test reports Cohen’s g once per discordant category pair, in a Cohen’s g by category pair table, rather than one averaged value. The two Cochran’s Q measures carry a superscript ⁿ naming the subject count each divides by – Cochran’s average φ² the informative subjects, Cohen’s w (Q) all of them – and a McNemar odds ratio computed with an empty discordant cell carries a ‡, the Haldane-Anscombe correction having kept it finite.

• Effect size confidence intervals – checked by default: an interval beside every measure, at the confidence level – inverted from the statistic’s own distribution where one exists, and a percentile bootstrap where none does, marked with a superscript b whose tooltip names the number of resamples the interval was actually taken over. The bootstrapped measures – Wilcoxon r, the matched-pairs rank-biserial r, Kruskal-Wallis ε² and η²_H, Kendall’s W, Kendall’s τ-b, Page’s ρ̄, Cohen’s g, Cochran’s average φ² and the explanatory measure ξ – resample over the Bootstrap replications setting, seeded by Bootstrap seed, and add that much runtime; unchecking the box skips them entirely. Two marks say more: on an unbalanced design ξ’s interval is marked conservative – wider than the estimator’s own sampling distribution – and a bootstrap whose every resample returns the same value reports – with a ! marker instead of a zero-width interval, and that estimate is left off the forest plot. A repeated measures ANOVA’s η² and generalized η² carry no interval – the point estimate alone. See interval for an effect size and percentile bootstrap.
• Standard errors – adds an SE column for the measures that have a closed-form one on their own scale: the d family – Cohen’s d and Hedges’ g, Glass’ Δ, Welch’s d and g, Cohen’s dz and Hedges’ gz, Cohen’s d_av and Hedges’ g_av, the one-sample d and g and the Algina-Keselman-Penfield robust d – and the rank-biserial r and Cliff’s δ. The variance-explained and χ²-family measures, the common language ES, the odds ratio and every bootstrapped measure carry none, and the cell stays blank there. Off by default.

Sums of squares #

For the factorial designs fitted through car::Anova – factorial ANOVA, ANCOVA, repeated measures factorial ANOVA, mixed ANOVA and mixed MANOVA – the partition of each effect’s variance on unbalanced data is chosen here; every other test uses Type III with no control, and ART-ANOVA partitions no sums of squares at all.

• Sums of squares – how each effect’s variance is partitioned when the cells are unequal – see Type II and Type III sums of squares; on balanced data the two agree, and the effects table notes which was used. In a repeated measures factorial ANOVA without covariates the between-subjects side of the model is intercept-only, where the two types reduce to the same test, so the note reads “Type III” whichever you picked; add a covariate and the choice takes effect. An empty cell of the between-subjects crossing cannot be fitted under Type III – Run comparison analysis stops with a message naming it – and can under Type II, whose main effects are tested by model comparison: the interaction row then comes back blank with a warning, and dropping or collapsing a factor, or batch analysis, are the other ways out.
• Type III (default) – every effect tested after adjusting for all the others, interactions included: order-invariant, the default of SPSS and most packages, and computed on the sum-to-zero contrast coding the module sets up.
• Type II – each main effect tested after the other main effects but not the interactions: more powerful when no interaction is present, and independent of the contrast coding.

Type II or Type III? With a negligible interaction Type II is the more powerful and less finicky choice; with a meaningful one Type III keeps each main effect interpretable in its presence and matches SPSS output – if in doubt, leave it on Type III. See Type II and Type III sums of squares.

Within-subjects approach #

For the three parametric families with a within-subjects factor – repeated measures ANOVA, repeated measures factorial ANOVA and mixed ANOVA – the route that tests the within-subjects terms is chosen here; a repeated measures ANOVA carrying a covariate is rerouted to a repeated measures MANOVA, which has no such route, and the control hides.

• Within-subjects approach – one route runs, and it drives the omnibus statistic, the significance verdict, the effect size, the post-hoc gate and the multiplicity pool together, so everything on the card belongs to one analysis – see univariate route and multivariate route. Between-subjects terms are tested on subject means and have no multivariate form, so they stay univariate either way, and the table says so. On a single-df within effect – two conditions – the two routes are numerically identical, sphericity being vacuous there. This is not Mixed MANOVA, which models several dependent variables jointly; the two answer different questions and can be combined.
• Univariate (ε-corrected) – the default: the classical repeated-measures F, its degrees of freedom corrected for sphericity by Greenhouse-Geisser or Huynh-Feldt, with Mauchly’s test and the ε columns on the card – see sphericity. It exists at any sample size.
• Multivariate (Pillai) – the within-subject contrasts tested jointly by Pillai’s trace, which assumes no sphericity at all – see Pillai’s trace. It needs more subjects than contrasts: a term that cannot support it keeps its univariate F, uncorrected for sphericity, and is named in the diagnostics, so a card can be multivariate for two effects and not for a third. Under this route the sphericity machinery – the ε columns, the corrected p-values, Mauchly’s table and its policy note – and the sums of squares are not carried, since they answer a question the route does not ask, and a note under the table says which columns belong to which route; the effect size is the partial η² recovered from the trace’s approximate F.

Heterogeneous slopes (regions of significance) #

Every test that crosses a covariate with a factor assumes the covariate’s slope is the same at every level – homogeneity of regression slopes, checked in the assumptions panel. When it fails, how far apart the groups are depends on where you stand on the covariate, so those tests close their card with a heterogeneous-slopes follow-up – per pair of levels, the slope difference with its test, the Johnson-Neyman boundaries and the share of the observations inside the region – see heterogeneous slopes results and the concept. A region is drawn for each pair whose slope test rejects at the assumption test significance level; one checkbox governs the rest:

• Show every region of significance – draws the region for every contrast, not only the ones whose slopes differ. It appears alongside the covariate role, so it is offered wherever a covariate can be assigned – ANCOVA, MANCOVA, factorial, repeated measures, repeated measures factorial and mixed ANOVA, repeated measures and mixed MANOVA; a repeated measures ANOVA with a covariate runs as a repeated measures MANOVA and keeps it. Off by default.

Discriminant analysis #

• Include descriptive discriminant analysis – names the combination of dependent variables the groups separate on – see descriptive discriminant analysis. Offered for MANOVA and MANCOVA alone, the two tests whose omnibus has a canonical decomposition to describe; it answers the question the multivariate test asks and the per-variable follow-ups cannot, and is added beside those follow-ups, not in place of them – see the results for the four tables it produces. Ticking it also reveals the Discriminant function plot checkbox under visualization, which draws what the tables describe. Off by default.

Correspondence analysis #

• Include correspondence analysis – decomposes the contingency table the test examined into dimensions and places its row and column categories in a shared space, where proximity reads as association – see correspondence analysis. Offered for the chi-square test of independence and Fisher’s exact test, the two tests that take an r×c table, and standing to them as discriminant analysis stands to MANOVA: not a second test but a description of what the omnibus found, with no p-value of its own – the test says an association exists, the decomposition says where it lives. The results are read in correspondence analysis results and drawn by the Correspondence map checkbox under visualization, which appears once this is checked; the Cochran-Mantel-Haenszel test’s stratified tables never reach it. Off by default.
• Decompose – which object is decomposed, shown once the box is checked.
• Each dependent variable against the grouping variable – the default: one r×c decomposition per dependent variable, attached to that variable’s test result – the direct follow-up to the test.
• All selected variables together – multiple correspondence analysis over every selected categorical variable jointly, the dependent variables and the grouping variable alike, as one run-level card – see multiple correspondence analysis. Offered once three categorical variables are selected – two dependent variables beside the grouping variable – and the run needs more complete cases than variables; a variable with a single realized level is dropped with a note, and two variables that split the cases identically are reported as redundant. Its inertia percentages are adjusted for the indicator coding and so not comparable with the two-way ones, as a note on the card says.

Two-way or joint? The two-way analysis reads each outcome against a clear grouping variable, the direct follow-up to the test; MCA asks how the whole set of categories organizes itself when the variables are on an equal footing – see multiple correspondence analysis.

Classification (ROC) analysis #

• Include classification (ROC) analysis – for the independent two-group tests that ask a location or ordering question – the independent samples t-test, Welch’s t-test, the Mann-Whitney U test, the Brunner-Munzel test and Yuen’s trimmed-means t-test – computes the area under the ROC curve and the classification metrics beside the inferential test: how well the dependent variable discriminates the two groups, and the cutoff above which a case would be classed as the higher-scoring group – see ROC curve and AUC. The option appears once an eligible test is selected and is off by default; the dispersion and whole-distribution tests, and every paired or repeated-measures design, are excluded. With three or more groups the curve is fitted per pairwise contrast, as the test itself is; a fit needs exactly two groups with at least four scored cases, and a variable whose AUC comes out below 0.5 has its groups swapped so that the higher-scoring group is always the predicted one, unless AUCs are being compared. The results are read under classification (ROC) results, and the curve is drawn by the ROC curve checkbox under visualization.
• Optimal threshold rule – how the cutoff is chosen; pick it before looking at the data, as with an equivalence bound, since choosing the rule that produces the threshold you wanted to see undermines the analysis. Whichever rule chose it, the threshold is chosen on the same cases it is scored on.
• Youden’s J (max sens + spec − 1) – the default: the cutoff that maximizes sensitivity + specificity − 1, weighing a missed case and a false alarm equally – see Youden’s J.
• Closest to (0, 1) – the point on the curve nearest the top-left corner, the perfect classifier – see closest to (0, 1).
• Cost-weighted – for errors of unequal cost: Youden’s rule with the specificity term weighted by the cost ratio and the prevalence – see cost-weighted. Both directions are reported as separate rows – one with the missed case as the costlier error, one with the false alarm – so you can pick the relevant one.
• Cost asymmetry ratio – shown under the cost-weighted rule: how many times worse one error is than the other, 3 by default. The field accepts values strictly greater than 1 – Run comparison analysis stops with a message otherwise – so to make the other error the costlier one, enter the reciprocal; the two rows cover both readings.
• Classification metrics at optimal threshold – checked by default: the per-threshold columns – sensitivity, specificity, PPV, NPV and accuracy – with bootstrap intervals that re-choose the cutoff inside each resample, over the Bootstrap replications setting, and the matching optimal-threshold markers on the ROC curve. Turn it off for the AUC summary alone – see sensitivity, specificity, PPV and NPV.
• Population prevalence – shown with the classification metrics: the share of the population that belongs to one group, for correcting the prevalence-dependent metrics – PPV, NPV and accuracy – to a known base rate when the sample’s class balance is a sampling artifact, case-control data above all – see prevalence. Empty by default, which bases them on the sample’s own group proportions; a value must lie strictly between 0 and 1, or the run stops with a message. Sensitivity, specificity and the AUC are unaffected; under the cost-weighted rule it additionally moves where the optimal threshold lands.
• Prevalence refers to group – appears once a prevalence is entered, with one select per factor the run will fit – every iterated grouping variable in batch mode – listing that factor’s levels: a prevalence is a statement about one of the two groups, so naming the level removes the ambiguity. Each defaults to the factor’s last level, the one an unflipped fit treats as the predicted group; a prevalence entered with no group named stops the run with a message.
• AUC confidence interval – the interval on the AUC, at the confidence level.
• DeLong – the default: closed-form, from the AUC’s own variance – see DeLong’s test.
• Bootstrap – a percentile interval resampled over the Bootstrap replications setting: slower, and the safer choice at a small sample.
• Compare AUCs (pairwise DeLong’s test) – appears when two or more dependent variables are selected: tests every pair of AUCs with DeLong’s correlated-AUC test, which accounts for the curves sharing the same subjects, the p-values adjusted as one family across every pair, group pair and batch factor under the global adjustment method – see AUC comparison. Because paired DeLong needs every curve on the same cases, rows missing on any selected dependent variable are dropped from every fit while it runs, so the per-AUC N is the shared subset, and every variable keeps one orientation, so a reverse-discriminating variable then legitimately shows an AUC below 0.5, with its cutoff chosen on the mirrored rule. Off by default.

Summary statistics #

Which descriptive statistics appear beside the test results, per group or condition, in the results tables and the CSV export; they do not change what the analysis computes. The Interpretation column’s direction phrase (“Group A > Group B”) needs a summary statistic to name a winner and gets one whether or not it is displayed – the mean for the location tests, the median for the rank-based ones, the standard deviation for the dispersion tests – so unchecking a box trims the table without dropping the direction from the verdict.

• Mean – on by default: each group’s arithmetic mean – see mean.
• Median – the middle value – see median; the rank-based tests read their direction from it.
• Mode – the most frequent value with its frequency and share, or No mode where every value is unique – see mode.
• Standard deviation – on by default – see standard deviation; the dispersion tests read their direction from it.
• Standard error – the standard error of the mean – see standard error.
• Confidence intervals of the mean – on by default: each group’s interval for its mean at the confidence level – see confidence interval – and, with it, the interval column of the difference itself (the estimate’s, for a one-sample test; the trimmed-mean difference’s, for the robust tests), the intervals in the pairwise tables and, in a design with a within-subjects factor and a subject ID, a second column per condition, the within-subject CI – the Cousineau-Morey interval the error bars draw, which supports comparing the repeated levels with each other but is not an interval for its own condition’s mean, as a note under the table says – see within-subject confidence interval. The same pair appears in the descriptive blocks of the factorial, mixed and multivariate cards, on the cells a within factor indexes.
• Percentiles (Q1, Q3, IQR) – the quartiles and their distance – see Q1, Q3 and IQR.
• Minimum and maximum – see min and max.
• Mean rank (for rank-based tests) – each group’s mean rank in the pooled ranking, the quantity the rank tests compare – see mean rank.
• Sample size – the cases each group’s statistics were computed from – see sample size.
• Joint frequency table – on by default, for the categorical tests: the contingency table of counts the test ran on – the χ² cells, McNemar’s discordant pairs, the CMH strata – and, under post-hoc tests, each pair’s own table; the goodness-of-fit test’s observed-against-expected table is shown regardless.

Include visualization #

• Include visualization – reveals the plot checkboxes, each with its own options, drawn under the results – see visualization for every plot. Which appear follows the design and the test: the distribution plots – box, violin, ECDF and mean and error bar – and the forest plot for the numeric families, the paired lines plot for a dependent or mixed design, the interaction plot once two factors are fitted; for the categorical tests the distribution and forest plots give way to the category proportion plot, except that the goodness-of-fit test draws its own observed-vs-expected chart instead and the one-sample proportion test keeps the forest plot, one Cohen’s h interval per variable. The discriminant function plot, the correspondence map and the ROC curve appear only once their analysis is enabled above. Off by default.

Error bar type #

• Error bars show: – shown whenever the mean and error bar plot or an interaction plot with error bars is on, and shared by both, so the two plots cannot disagree; the caption under each plot names what its bars span. In a purely within-subjects design the standard-error option drops out – a between-subject SE is not a meaningful bar there – and the confidence-interval option reads Within-subject confidence interval; in a mixed design every option stays, but a cell drawn from a within-subject interval draws that interval whatever a CI or SE request said, only the SD of the observations surviving as asked.
• Confidence interval of the mean – the default: the precision of the mean, at the confidence level – see confidence interval. Where the design has a within-subjects factor the bar is the Cousineau-Morey within-subject interval – the one the tables print – which supports comparing the repeated levels with each other, though two of its bars just touching is not a test.
• Difference-adjusted within-subject confidence interval – shown only where the design has a within-subjects factor: the within-subject interval rescaled by $\sqrt{2}/2$ so that two bars overlap exactly when the difference between them is not significant – see difference-adjusted interval.
• Standard error of the mean – ±1 SE, roughly two-thirds of the CI’s half-width – see standard error.
• Standard deviation – ±1 SD of the observations: the spread of the data, not the precision of the mean, so non-overlapping SD bars say nothing about whether two means differ – see standard deviation. Pick it to show how variable the data is, and CI or SE to show what the comparison rests on.

Checking assumptions #

Click Check assumptions to run, before any test, the checks the current design and selection call for, and read them in an Assumption test results card with three parts: a summary table with one row per assumption, the test recommendations that read those rows against the sample, and the detailed results behind each row. Which rows appear follows the roles you have set – a variance check needs a grouping variable, the covariate checks a covariate, the categorical checks a categorical dependent variable. The tests among the checks are judged at the assumption test significance level, Box’s M at a fixed α = 0.001, and the count rules at their own thresholds. Dependent and mixed designs need a Subject ID here as for the run itself, and under Analyze each variable separately one card is drawn per factor, titled with that factor’s name and checked on its own partition.

Summary table #

One row per assumption the design calls for, in the order below; a row a design does not need is absent rather than blank.

• Assumption – the condition the row checks, named with the test that checks it – see what an assumption is. The note beside it says what was tested and on what, which matters because normality in one design means the values within each cell and in another the residuals of a model.
• Status – Pass, Fail, Skipped when the check could not run on this selection – the note says why – or N/A when it ran and nothing was assessable. In a detailed table the same column carries a named verdict instead – Equal variances, MV normal, Symmetric, Sphericity assumed, Homogeneous, Linear, Independent, Adequate and their opposites – or the reason none was given: Insufficient data, Singular covariance, Not applicable, Not binary, Only one category, Could not compute. A failure is a flag the recommendations weigh against the sample, not a verdict on a test – see Status and Not assessed.
• Notes – what the verdict rests on: the count of tests passed or cells failed out of those assessed, the largest |r|, the reason a check was skipped, and, for a per-cell family in a wide design, whether the failures stayed within what chance produces across that many tests – in which case the row says so and nothing is demoted.
• Normality (Shapiro-Wilk) – Shapiro-Wilk on what the chosen family assumes normal: each dependent variable within each group – each cell of the crossing in a factorial design – in an independent design; the paired differences in a two-condition dependent design; the residuals of the within-subjects model with three or more conditions; the residuals of the covariate-adjusted model when a covariate is selected; each between × within cell in a mixed design; and each variable’s own values in one-sample mode. The note counts the cells that passed out of those tested – a cell under 3 or over 5,000 values is left untested – and a failure demotes the parametric tests only where a failing cell is under the small-sample advisory floor; above it they stay recommended with a central-limit caveat. See normality and the Shapiro–Wilk test.
• Outliers (studentized residuals) – the residuals behind the normality rows, each studentized against the model’s own error and Bonferroni-tested, so a flag marks an observation more extreme than a normal sample of this size produces rather than one past a fixed cutoff; the note counts the flagged observations out of those assessed. A flag never demotes a test: the tests on means carry a caveat to inspect the flagged cells before reading the result, and the F-test and Bartlett’s a sharper one, since a variance follows an extreme value far more steeply than a mean. The row reads Skipped on the per-factor normality fallback, which has no single residual pool. See studentized residual and outlier.
• Multivariate normality (Mardia’s test) – with two or more numeric dependent variables: Mardia’s skewness and kurtosis per group – on the covariate-adjusted residuals when a covariate is selected, and on each subject’s (variable × condition) responses in a dependent design, as one group, or a mixed one, per between-subjects cell; either statistic rejecting in any group fails the row and demotes the MANOVA family and Johansen’s. The row reads N/A when no group had enough complete cases – a group needs more cases than variables – and those tests then stay recommended with a note that the assumption is unverified; a group that could not be assessed beside ones that could adds the same note. See multivariate normality and Mardia’s test.
• Homogeneity of variance (Levene’s test, Brown-Forsythe) – the median-centred Levene’s test per dependent variable across the groups – the cells of the crossing in a factorial design – in an independent design; across the between-subjects groups on each subject’s mean over the conditions in a mixed one; and on the residuals of the covariate-adjusted model when a covariate is selected. The note counts the variables with unequal variances. A failure demotes the tests that pool one variance – Student’s t, one-way and factorial ANOVA, ANCOVA – and leaves the Welch tests as the recommended alternatives; Mann-Whitney and Kruskal-Wallis stay recommended with a caveat, since unequal spreads break their location-shift reading; in a mixed design the failure reaches mixed ANOVA under the balance rule of the row below. See homogeneity of variance and Levene’s test.
• Homogeneity of variance by condition (Levene’s test, Brown-Forsythe) – mixed designs only: the same test across the between-subjects groups within each within-subjects condition, one row per variable × condition, over the subjects complete across every condition – the assumption the mixed ANOVA’s F-tests rest on. The note counts the failing cells, and a wide family counts as a violation only once its failures outrun chance. A violation demotes mixed ANOVA when its cells are unequal in size – the largest more than 1.5× the smallest – and leaves it recommended with a caveat when they are roughly equal, since a balanced mixed ANOVA tolerates unequal group variances; ART-ANOVA keeps a caveat either way, because the alignment corrects no variance difference.
• Sphericity (Mauchly’s test) – dependent and mixed designs whose within-subjects factor has three or more levels: Mauchly’s W per dependent variable – one row per within-subjects term when there are several, named in an Effect column – from the same wide fit as the repeated measures ANOVA, adjusted for the between-subjects factor of a mixed design and for a covariate, and fitted on the combined factor when the within-subjects crossing is incomplete. The detailed table adds the n of subjects the fit kept and the four correction columns below. The note counts the variables violating it; with no term of three or more levels the row reads Skipped as not applicable, since sphericity then holds by construction. A violation never demotes: repeated measures ANOVA, repeated measures factorial ANOVA and mixed ANOVA stay recommended with a caveat to read the Greenhouse-Geisser or Huynh-Feldt p-value, which their results table already takes its verdict from; a Mauchly test that could not compute adds a caveat that sphericity is unverified. See sphericity and Mauchly’s test.
• Homogeneity of covariance matrices (Box’s M) – with two or more numeric dependent variables: Box’s M across the groups – the cells of the crossing – at a fixed α = 0.001 whatever the assumption test significance level; on the covariate-adjusted residuals when a covariate is selected, and in a mixed design, with any number of dependent variables, on each subject’s (variable × condition) responses across the between-subjects groups – multi-sample sphericity, an assumption of the univariate mixed ANOVA too. A failure demotes the pooled multivariate tests – MANOVA, MANCOVA, repeated measures and mixed MANOVA – only when the cells are unequal in size, the largest more than 1.5× the smallest, steering MANOVA and MANCOVA to Johansen’s heteroscedastic MANOVA; with roughly equal cells they stay recommended with a caveat, and mixed ANOVA follows the same split. The energy test is demoted on any failure, since it rejects on unequal spread as readily as on unequal means – unless multivariate normality has failed too, when Box’s M can establish nothing and the demotion softens to a caveat. The row reads Skipped with the reason when the test could not run: a group with no more cases than variables, a singular covariance matrix, or, in a mixed design, a group with fewer than 5 subjects complete across the conditions. See homogeneity of covariance matrices and Box’s M test.
• Multicollinearity – with two or more numeric dependent variables: the correlation of every pair on the error covariance – the residuals once the groups and any covariate are fitted, or once subject and condition are in a dependent or mixed design – flagged at |r| above 0.90; the note gives the flagged count or the largest |r|, and the detailed table lists the flagged pairs. A flag demotes the tests that invert a pooled covariance matrix – the MANOVA family and Johansen’s – and leaves the energy test, which computes distances instead; a check that could not be evaluated leaves them recommended with a note that it is unverified. See multicollinearity.
• Homogeneity of regression slopes – with a covariate: the covariate × factor interaction, tested per dependent variable × covariate – against the grouping variable in an independent design, the condition variable in a dependent one, and both in a mixed one. A failure demotes the covariate-bearing tests – ANCOVA, MANCOVA and factorial ANOVA with a covariate, the repeated measures family, or the mixed family – with a reason naming what the card runs in their place: each pair’s region of significance. See homogeneity of regression slopes.
• Linearity (Covariate–DV) – with a covariate: an F-test on a squared covariate term added to the model within the groups, per dependent variable × covariate – at the subject level in a dependent design, and against the between-subjects side alone in a mixed one; curvature fails the row and demotes the covariate-bearing tests. See linearity.
• Covariate independence – with a covariate and a grouping variable: a one-way ANOVA of each covariate on the groups – on one row per subject in a mixed design; a covariate that differs across the groups fails the row and demotes the covariate-bearing tests as confounded with the treatment. A pure within-subjects design has no such row, since a covariate that takes one value per subject cannot vary with the condition. See covariate independence.
• Covariate collinearity – with two or more covariates: the same |r| > 0.90 check within the covariate set; a flagged pair is a caveat on the covariate-bearing tests rather than a demotion – the model still fits, and which covariate to drop is yours to decide – and the detailed table lists the flagged pairs.
• Covariate assumptions – the one row the three covariate checks collapse into, reading Skipped with the reason, when their model could not be fitted at all.
• Expected frequencies – independent designs with a categorical dependent variable: Cochran’s rule on the expected counts of each variable’s table against the grouping variable – the crossing, under the combined strategy – with no cell expecting fewer than 1 and no more than a fifth of the cells fewer than 5. With a stratifying variable the rule runs over the per-stratum tables the Cochran-Mantel-Haenszel test pools – the Mantel-Fleiss criterion on a 2 × 2 × K table – and the note warns about the CMH approximation instead of steering to Fisher’s exact test. The row demotes nothing: the categorical tests are in neither recommendation list, and the chi-square test itself warns under its table when the rule fails. See expected frequencies.
• Expected frequencies (goodness-of-fit) – one-sample mode with a categorical dependent variable: the same rule on the counts your expected distribution implies – N times each weight’s share, or a uniform split while you have entered nothing. When it fails the goodness-of-fit test switches its p-value to Monte-Carlo simulation on its own, and the note says so; a hypothesis with fewer than two categories or a weight that is not positive leaves the variable untested, the note asking for a positive value in every category – the same weights Run comparison analysis refuses.
• Discordant pairs (McNemar / Stuart-Maxwell) – dependent designs with a categorical dependent variable, when McNemar’s or Stuart-Maxwell’s test is selected – or two conditions with no paired categorical test selected: the pairs whose two conditions disagree, counted per variable – over every condition pair when there are more than two, the weakest pair reported – with fewer than 25 failing the row. With two categories the note says the exact binomial test takes over on its own below that count; with more, that the approximation may be unreliable, since Stuart-Maxwell has no exact form. Nothing is demoted. See discordant pairs.
• Informative subjects (Cochran’s Q) – dependent designs with a categorical dependent variable, when Cochran’s Q is selected – or three or more conditions with no paired categorical test selected: the subjects whose response varies across the conditions, with fewer than 4 of them – or their number times the condition count at 24 or under – failing the row, and a variable that is not binary reported as such. Nothing is demoted. See informative subjects.
• Symmetry – one-sample and two-condition dependent designs: a sign-flip permutation test on the skewness of each variable’s values, or of the paired differences – the assumption the Wilcoxon signed-rank test makes in place of normality, so a symmetric heavy-tailed sample fails the normality row and passes this one. A failure never demotes: the signed-rank test stays recommended with a caveat steering to the sign test or the paired sign test, which assume nothing about shape. The row is absent when no variable had the 6 values the test needs. See symmetry.

Test recommendations #

Every test the design offers is listed once – as recommended, or as not recommended with its reasons – with each verdict above read against the sample rather than taken at face value. The categorical tests are in neither list, since their assumption is the expected frequencies row alone.

• Recommended tests – the tests whose assumptions hold, or whose failure the sample covers, each with its caveats in muted text after a dash. A caveat means the test is still safe with something to keep an eye on: normality failed but every group – every sample, or the paired differences – clears the Small-sample advisory floor (N), so the test is robust by the central limit theorem, and raising or lowering the setting moves that line; a wide family’s failures stayed within chance; sphericity failed, so read the corrected p-value; extreme observations were flagged; a check could not be assessed – normality, homogeneity of variance, multivariate normality, sphericity, covariance matrices, multicollinearity – so the assumption is unverified, cautionary when the sample was too small to test and reassuring when it was too large; unequal variances under Mann-Whitney or Kruskal-Wallis, with the test to prefer; a balanced design absorbing a Box’s M or a by-condition Levene failure; collinear covariates. The tests that are never demoted carry a standing caveat instead – the sign tests always, since they read only directions; the trimmed-means tests, Fligner-Killeen and Johansen’s while the assumption they relax still holds, as the lower-powered fallback – and ART-ANOVA and repeated measures factorial ART always say that they assume equal variances across every cell, the within-subjects levels included, and correct no departure, the spread of the cell standard deviations below being what shows how far off they are.
• Not recommended – each demoted test with its reasons: a violated assumption the sample does not cover – Normality assumption violated, or of model residuals, of the between × within cells, of the per-factor marginals, naming what was tested; Homogeneity of variance violated; Multivariate normality violated; the Box’s M and multicollinearity verdicts; Covariate confounded with treatment; Linearity assumption violated; and the slopes reason that names the region of significance the card reports instead – or a design the test cannot run on: Requires a two-group comparison, Requires a factor with two or more levels, Requires two or more grouping factors, Requires two or more within-subject factors, Requires a single ordinal factor with at least 3 levels for the trend tests, and Not assessed – specify the full model to evaluate while a role the test needs is still empty – the same requirements Run comparison analysis enforces (see choosing a test).

Detailed results #

One table per check, in the order of the summary rows, with a Status verdict per row coloured by outcome – green for a pass, red for a failure, amber for a cell too small or too large to test or a low-frequency flag, grey for a check that did not apply or could not compute – and a note above a table naming its basis wherever that is not the raw values: a model’s residuals, subject means, the subjects complete across every condition. A check that could not run at all shows its reason in place of a table.

• Normality tests (Shapiro-Wilk) – variables × cells, each cell carrying W, its p-value, n – a subject count where a within-subjects model’s residuals were tested – and the cell’s shape moments; a cell under 3 values reads n < 3, one over 5,000 n > 5000, both untested but with their moments shown. A note above names the residual tested – the within-subjects model’s, the covariate-adjusted model’s, or the per-cell values over complete subjects – and in a mixed design the columns are the between × within cells. For the full test battery (D’Agostino-Pearson K², Jarque-Bera, Anderson-Darling), run the same variables through Normality analysis. See the Shapiro–Wilk test.
• Skewness – with Excess kurtosis, the muted lines at the foot of a normality cell: how far the cell’s values lean to one side, and how heavy their tails are against a normal curve’s. Both are 0 for a normal distribution and unmoved by sample size, unlike the p-value above them, which rejects trivial departures at large n and misses real ones at small n – so they say how far from normal a cell is where W says only whether. See skewness and kurtosis.
• Sample – the one column of the normality and symmetry tables in one-sample mode, each variable’s own values; Residuals the one column where a model’s residuals were tested. In the outlier and Mardia tables the same words name the cell, and All subjects a within-subjects design’s one pooled group.
• Between-subjects – with Within-subjects, the column groups of the normality table when a mixed design’s crossing exceeds 50 cells or could not be computed and the table falls back to each factor’s own levels: the between-subjects rows are tested on each subject’s mean over the conditions, the within-subjects rows on per-subject means at each level of the factor, so their n counts subjects and no subject is counted once per condition; a note above the table says which fallback applied. The Q-Q plots and the outlier row are withheld on this fallback, whose rows overlap. See between-subjects factor and within-subjects factor.
• Residual Q-Q plot – one per dependent variable under the normality table, drawn on the same residuals its rows were tested on, every cell centred on its own mean and pooled – so departures in opposite directions can cancel in the pooled picture; read it beside the per-row numbers, not instead of them. A variable with more than 5,000 residuals is drawn from a 5,000-point subsample taken in rank order – the same curve at lower resolution, not a random sample – and a variable with fewer than 5 gets no plot. See Q–Q plot.
• Flagged – in the outliers table, which lists the flagged cells alone: how many of the cell’s n assessed observations the test flagged.
• Most extreme residual – the cell’s largest externally studentized residual – the observation’s distance from its cell mean in units of the model’s error, with the observation itself left out of that error – and p (Bonferroni) its two-sided p-value multiplied by the number of observations in the variable’s pool, a flag falling where that stays under the assumption test significance level.
• Skewness χ² – Mardia’s multivariate skewness read against a χ² with p(p + 1)(p + 2)/6 degrees of freedom for p variables, its p-value under Skew. p; Kurtosis Z the multivariate kurtosis read as a two-sided z, its p-value under Kurt. p. The Group column names the between-subjects cell, Residuals when a covariate was adjusted for, or All subjects for a within-subjects design’s one pooled group.
• Symmetry (skewness sign-flip test) – one row per variable with the sample skewness g₁ under Skewness, n, and a p-value from 2,000 sign-flip replicates of the deviations from the median under a pinned seed; Sample or Difference names what was tested. Verdicts Symmetric, Asymmetric, or n < 6 where the test cannot reach a conventional α. See symmetry.
• Spread of the cell standard deviations – shown when an ART test is among the recommendations: one row per dependent variable with the number of Cells, the Smallest SD and Largest SD among them, and their ratio under Largest / smallest, computed on exactly the sample the ART fit uses, so it matches the numbers beside the ART-ANOVA results. It is a diagnostic without a verdict – no threshold is offered, because how much a ratio matters depends on the cell sizes and on whether the spread runs across the between-subjects groups or the within-subjects levels – and a cell with no variance keeps its place in the SD columns and leaves the ratio blank. See the aligned rank transform and the variance ratio.
• ε (GG) – the Greenhouse-Geisser estimate of how far the variances of the pairwise condition differences depart from equality – 1 when sphericity holds, down to 1/(k − 1) for k levels – which the results table multiplies both degrees of freedom of the F by. See the Greenhouse–Geisser correction.
• p (GG) – the p-value of the effect’s F read with both degrees of freedom multiplied by ε (GG) – the one to report when ε (GG) is below 0.75.
• ε (HF) – the Huynh-Feldt estimate of the same quantity, a little larger than ε (GG) and displayed capped at 1. See the Huynh–Feldt correction.
• p (HF) – the p-value under the Huynh-Feldt correction – the one the results table’s verdict reads when ε (GG) is 0.75 or more.
• Box’s M test – the statistic as an approximate χ² (approx.) with its df and p-value, judged at the fixed α = 0.001 the note under it repeats; a note above names the basis when it is not the raw group matrices – the covariate-adjusted residuals, or each subject’s (variable × condition) responses in a mixed design. A test that could not run shows its reason in place of the table.
• DV – the dependent variable of a covariate row, one row per dependent variable × covariate, and Covariate the covariate the row tests – see covariate.
• Across – in a mixed design’s slopes table, which side the row tests: Groups (between) for the covariate × grouping interaction on one row per subject, Conditions (within) for the covariate × condition interaction; a between-subjects pass that could not be fitted leaves the within rows alone, with a note saying why the linearity and independence tables are missing.
• F (interaction) – the F-test of the covariate × factor term, with its degrees of freedom and p-value; verdicts Homogeneous and Non-homogeneous, or Not applicable and Could not compute. The linearity table carries the F of the squared term and the independence table the F of the one-way ANOVA the same way.
• df1 – with df2, the numerator and denominator degrees of freedom of the F beside them: in the Levene tables the groups minus one and the observations minus the groups; in the slopes, linearity and covariate-independence tables the term’s coefficients and the model’s residual. A fractional value is shown to one decimal.
• Covariate 1 – with Covariate 2, the flagged pair of the covariate collinearity table, and r their correlation.
• Min. expected – the smallest expected count in the variable’s table, or across its strata – the count Cochran’s rule wants at 1 or more.
• Cells < 5 – the number of cells whose expected count is below 5, and % cells < 5 their share of the table, the share the rule caps at a fifth; Total cells the cells counted, over every stratum when there is a Strata column, and Categories the category count of a goodness-of-fit table, where each category is a cell, or of a paired variable. Verdicts Adequate and Low frequencies, Only one category where the variable has nothing to test, and Expected distribution incomplete where the weights entered are not all positive.
• Discordant pairs – the table under that heading and its first count: per variable, the pairs whose two conditions disagree – the smallest count over the condition pairs when there are more than two, which the note says – beside Complete pairs, the pairs with a value in both conditions, and Categories; verdicts Adequate and Too few, or Only one category.
• Informative subjects – the count column of the Cochran’s Q table: the subjects whose response varies across the conditions, beside Complete subjects, the subjects with a value in every condition; verdicts Adequate, Too few, and Not binary for a variable with more or fewer than two categories. See informative subjects and complete subjects.

Reading results #

Click Run comparison analysis to run the test. The setup is checked before any R call – a missing role, a test the design cannot run, a stale direction – and a refusal names what to change (when a test refuses to run). The card is titled Comparison analysis: {test} ({factor}) – One-sample comparison: {test} in one-sample mode, its own title under a batch or pseudo-factorial run – and holds, in order, whichever of these the run produced: the Overall test results of a one-factor test with the tables that follow it – the contingency tables, where the distributions separate, Mauchly’s test – and its post-hoc comparisons; the test errors; the family cards – factorial, mixed, ART, multivariate and ANCOVA; the pairwise comparisons; the classification analysis; the correspondence analysis; and the plots. Whatever a run decided on the way – a subject dropped, a branch switched, a fallback taken – is disclosed in a note under the table it concerns, so nothing happens silently: read the notes before the numbers.

Test errors. The section a family fit that failed outright lands in – a factorial, mixed, ART, covariate-adjusted or multivariate test – headed {n} test(s) failed: with each error message, so a card that could not be drawn is never silently absent. A one-factor test’s failures are listed under its own table instead, {n} variable(s) failed analysis:, grouped by message with the failed variables under each, and All variables failed analysis replaces the table when no row survived; a package’s own message is quoted as it came.

Diagnostics. The box a family card prints under its source table when the fit decided something on your behalf: an effect size the package could not compute, a between-cell imbalance with the range of cell sizes, subjects dropped listwise for a missing within-subject cell or covariate value, a significant covariate × within-subject interaction, a higher-order interaction left without contrasts, a term that could not be tested multivariately, a post-hoc term that failed, covariates an ART test ignored, or ART’s alignment check failing. Read it before the table it sits under.

Standard tests (t-test, ANOVA, Kruskal-Wallis, etc.) #

Overall test results. One row per dependent variable for every test that is not a family card – the t-tests and their rank, robust and sign siblings, the one-way ANOVAs, the dispersion and whole-distribution tests, the trend tests, the categorical tests and the one-sample tests – with the columns the test and your options call for: the summary statistics per group or condition, the estimate and its interval where the test has one, the statistic, its degrees of freedom, the p-value and the columns an adjustment or an equivalence direction adds, the effect size with its interval and, when interpretation is on, the verdict. A two-sample test run on three or more levels prints no row here – its pairs are the pairwise comparisons.

The per-group columns come first, one set per group or condition headed by its level – {group} N, {group} Mean, {group} SD, {group} {level}% CI, … – bare N, Mean, … in one-sample mode, each read under the summary statistic that enables it; a categorical dependent variable carries N alone, since its levels have no mean.

• Variable – the dependent variable the row tests – or, on a multiple correspondence analysis card, the variable a category belongs to. In a multi-variable run a group empty on one variable keeps its row, with that group’s columns blank, so the gap is visible rather than dropped or filled with placeholder numbers.
• {group} {level}% within-subject CI – a second interval per condition in a design with a within-subjects factor and a subject ID: the Cousineau-Morey interval, which removes the between-subject spread and so supports comparing the repeated levels with each other, and is not an interval for that condition’s own mean – the ordinary interval beside it is, as the note under the table says. See within-subject confidence interval.
• Category – on the one-sample proportion test’s row, the category counted as the success – the choice made under Category counted as a success, repeated here because on a variable with more than two categories nothing else on the card says which share was tested. See success category.
• μ₀ – the value the row’s variable was tested against, in one-sample mode: the Reference value, or its own under Set per variable; headed π₀ on the proportion test’s row, where it is the hypothesised proportion.
• Estimate – the one-sample tests’ location estimate, the quantity the interval beside it brackets: the mean for the t-test, the Hodges-Lehmann pseudo-median for the Wilcoxon signed-rank test, the median for the sign test, and the observed share, headed p̂, for the proportion test.
• Difference – the two-group tests’ own estimate, each test reporting what its machinery estimates: the mean difference for the t-tests, the difference of trimmed means for Yuen’s, and the Hodges-Lehmann shift – the median of every between-group pairwise difference – for Mann-Whitney, and its paired form, the pseudo-median of the differences, for the Wilcoxon signed-rank test. The header names the two levels in the order they were subtracted, Difference ({first} − {second}), so a sign is never ambiguous, and the verdict’s direction line reads from this estimate.
• {level}% CI of Difference – the interval of that estimate, shown with Confidence intervals of the mean: headed {level}% CI of {first} − {second}, … of trimmed-mean difference (…) for the robust tests, whose estimand is the trimmed-mean difference, and … CI of estimate in one-sample mode. It is two-sided even under a one-tailed direction – it describes the size and precision of the effect while the one-tailed p carries the decision – and an interval excluding zero goes with a two-tailed p below α, as the interval and the p-value explains; under an equivalence direction it is built at 1 − 2α, and the header says so. Under a p-value adjustment every interval on the card stays at the nominal level, as a note under the table repeats.
• Statistic – the column headed by the test’s own symbol, with the significance stars of the row’s decision p: t, F – F* for Brown-Forsythe – H, U, W for Brunner-Munzel, Ty for Yuen’s, D for Kolmogorov-Smirnov, T.AD, J, L, Z for Cochran-Armitage, K² for Bartlett’s, χ², Q, I and JSD in bits, and V for the Wilcoxon signed-rank tests – printed as V/maxV, the rank sum out of the most it could reach, as the sign tests’ S is printed as positive signs out of the non-tied n and the proportion test’s x as successes out of n; under the multivariate route a repeated measures ANOVA’s V is Pillai’s trace, with that trace’s approximate F beside it. Fisher’s exact test has no statistic and no column; a test whose exact branch reports none – McNemar’s below 25 discordant pairs, Brunner-Munzel’s permutation branch, the Cochran-Mantel-Haenszel exact conditional test – leaves the cell blank, and a note says which branch ran. The header’s help opens the test’s own entry under Choosing a test; the effect-size column beside it is headed by the measure’s own symbol, its help opening that measure’s definition, and is read under Effect sizes. See test statistic.
• df (effect, error) – the two degrees of freedom of an F, the effect’s and the error’s, on the one-way ANOVA rows; a plain df heads the t-tests’ single value – Welch’s and Yuen’s fractional, to one decimal – and the χ² tests’ – see df. The trend tests, the sign tests and the exact branches report none, and the column is absent when no row has one.
• df (num, denom) – the F-test of equality of variances’ pair, the numerator’s and the denominator’s, n₁ − 1 and n₂ − 1 – not an effect and an error.
• df (GG) – a repeated measures ANOVA’s two degrees of freedom multiplied by ε (GG), the pair a Greenhouse-Geisser-corrected F is reported against – F(3.7, 744.7) where the uncorrected pair was 5 and 995 – between the ε (GG) and p (GG) columns the univariate route prints on a factor with three or more levels, with Mauchly’s test in its own table below, one row per variable; under the multivariate route the six sphericity columns and the Mauchly table give way to the V column. See the Greenhouse–Geisser correction.
• df (HF) – the same pair under the Huynh-Feldt ε, capped at 1 as its column is – see the Huynh–Feldt correction. The verdict reads the corrected p Girden’s rule picks – GG while ε (GG) is below 0.75, HF from 0.75 up – whatever Mauchly’s test says, and tags it GG-corrected or HF-corrected.

The p column is the test’s p-value, formatted per your p-value settings – see p. Under a p-value adjustment in addition mode a p (adj) column follows it – a sphericity-corrected row’s cell is labelled p (adj, GG) or p (adj, HF), since the pool took the corrected p – and in replacement mode the adjusted value stands in the p column itself; see p (adj). The effect-size columns come next – the measure’s symbol, its CI and, on request, its SE – read under Effect sizes, which also explains the ⁿ, ‡, b and ! marks a cell can carry; a blank cell with a ! says a group has fewer than two observations, so no standardized difference exists.

• Δ₂ – Glass’ Δ standardized by the second group’s SD, with its own Δ₂ CI and Δ₂ SE, beside the Δ of the main column standardized by the first group’s – see Glass’ Δ.
• g – on a t-test row the bias-corrected d you chose, Hedges’ g or Welch’s g, whose interval is the uncorrected d’s, as a note under the table says; on a McNemar or Stuart-Maxwell row, and in the Cohen’s g by category pair table, Cohen’s g, the measure of marginal change.
• Interpretation – the verdict, in three parts. The decision at the significance level – read from the adjusted p when one was applied, and from the corrected p on a repeated measures row, tagged GG-corrected or HF-corrected – worded for the hypothesis the test states: Significant difference, … association, … difference in variances, … between distributions, Observed distribution differs from expected, Observed proportion differs from π₀; Significantly greater (one-tailed) or No significant directional difference under a one-tailed direction; Significant increasing trend and its kin for a trend test; the equivalence verdicts; Undetermined where the p could not be computed. Then the effect’s magnitude against the measure’s benchmarks – Δ₁ and Δ₂ each for Glass’ Δ. Then a direction line – Group 1 < Group 2 – read from the test’s own estimate where it has one and otherwise from the family’s summary statistic, suffixed (SD) for the dispersion tests and (Median) for the whole-distribution tests, which order the groups by it without testing it; {variable} < μ₀ or p̂ < π₀ in one-sample mode; Direction confirmed: or Unexpected direction: on a significant one-tailed test. The Cochran-Mantel-Haenszel row prints no direction line – the pooled table’s can point against the common odds ratio, which carries the direction – and a one-tailed Fisher row keeps the plain line, the note under the table naming the odds it tested. See interpretation.

Notes under the table say what the run did that the columns cannot show. Run-wide: the trim level of a trimmed-means test, γ = 0.2 (20% from each tail), with a warning above 0.25; that I and JSD are in bits against a ceiling set by the design; that the intervals are not adjusted for multiplicity when the p-values are; that the within-subject interval is the Cousineau-Morey one; that a Hedges’ g interval is the uncorrected d’s; that a repeated-measures ω²p is the noncentrality plug-in its interval inverts, a little larger than effectsize’s and SPSS’s; that the within-subjects effect was tested by Pillai’s trace; that every within-subjects effect is corrected whatever Mauchly’s test says. Per variable: subjects excluded for missing a value in some condition, and subjects with more than one row per (subject, condition) cell, averaged before the test – counted once for the run; observations equal to μ₀, or pairs with a zero difference, dropped as ties by the sign tests and, below 50 pairs, kept and ranked by the Wilcoxon tests; an exact branch that ignored the continuity correction, or a correction that had no effect on a table larger than 2×2; a median interval whose discrete coverage is not the requested level, or is unbounded; an odds ratio not computed because the table was not 2×2 – 2×2×K under CMH – and, where it was, which odds it divides by which; expected counts below 5; the trend scores a Cochran-Armitage test fitted; a p-value from Monte-Carlo simulation or permutation with its replicate count, flagged when it sits at the simulation’s floor; the Breslow-Day test, sparse or zero-cell strata and strata dropped for holding a single observation under CMH; a singular or rank-reduced Stuart-Maxwell matrix; the category coded as the success; the odds a one-tailed Fisher test ordered; a within-subject effect that could not be tested multivariately; Brunner-Munzel’s exact branch; a Kolmogorov-Smirnov p that had to be asymptotic.

A trend test’s row carries its own statistic – Jonckheere-Terpstra’s J, Page’s L, Cochran-Armitage’s signed Z, positive for a rising rate – and no degrees of freedom; its p follows the trend direction you chose and its verdict names it, Significant increasing trend, decreasing or monotonic. Jonckheere-Terpstra takes a seeded permutation p on ties or at n ≤ 100, with a note giving the replicate count from the permutation replications setting; Page’s L takes its exact null distribution below 16 conditions and 21 blocks when no block holds tied ranks, and the tie-corrected normal approximation otherwise. Kendall’s τ-b, Page’s ρ̄ and Cohen’s w are their effect sizes, and the summary columns run in the factor’s ascending order, so the trend reads left to right.

If any variables fail, an error summary under the table groups the failures by message.

Where the distributions separate #

Where the distributions separate. The Kolmogorov-Smirnov card’s second table, one row per dependent variable – and, on a pairwise expansion, one per pair inside a collapsible Where the distributions separate: {variable} panel – breaking D into the two one-sided gaps behind it, so that the card says which way the distributions part and where along the scale, which D alone cannot. A note beneath spells the orientation out – D⁺ is the amount by which the first group’s curve runs above the second’s – and on a two-group run the winning gap is marked on the ECDF plot as a vertical drop between the curves at its value. The test itself stays two-sided; the note under the results table says that this is conservative for a directional reading, since in the tail $P(D>d)\approx 2P(D^{+}>d)$, so a two-sided rejection also supports the one-sided claim in the direction of the larger gap. See Kolmogorov-Smirnov D.

• D⁺ and D⁻ – the largest amount by which the first group’s cumulative curve runs above the second’s, and the largest by which it runs below; D is the larger of the two – see D⁺ and D⁻.
• Value at D⁺ and Value at D⁻ – the value of the dependent variable at which each gap is reached: a separation in the lower tail is a different finding from one at the median.
• Larger gap – which of the two is D, the statistic tested above; – when they tie.

Equivalence test results #

Under an equivalence direction the results table opens with a bounds note and adds the equivalence columns after the two-sided p. The note names the direction – Equivalence (TOST), Non-inferiority (higher is better), Superiority (lower is better), Minimal effect (MET) – and the bound: a raw Δ as entered; a standardized Δ as the entered value with the spread that converted it – Welch’s d standardized by the root mean square of the two SDs, Cohen’s dz standardized by the SD of the differences, robust standardized units by a median absolute deviation for the rank tests, or the SD standing in where the MAD was zero – and its raw value, listed per variable when several variables’ raw bounds differ; in one-sample mode the shifted nulls μ₀ ± Δ themselves, on the scale of the estimate’s interval. Two more lines say that the difference interval is built at 1 − 2α, so that an interval inside the bounds is the same decision as the test rejecting at α – see the (1 − 2α) interval – and, for a standardized Δ, that the bounds are held fixed at a conversion which is itself an estimate.

• p (two-sided) – what the plain p column becomes in equivalence mode: the ordinary two-sided test of “do the groups differ?”, which has nothing to do with whether they are equivalent. Read the equivalence decision from p (equiv) or p (MET) and the verdict, never from this column – a difference can be both statistically significant and practically equivalent, and the verdict says so when it is.
• p (lower) and p (upper) – the two one-sided tests, one per bound, shown for TOST and MET, which test both bounds; non-inferiority and superiority test one bound and show its p alone.
• p (equiv) – the equivalence decision: the larger of the two one-sided p-values, since both bounds must be cleared – see TOST; headed p (MET) for the minimal-effect test, where it is the smaller, either bound being enough – see minimal-effect test. The verdict follows it – Equivalent (within Δ = 0.5), Non-inferior (Δ = 0.5), Superior (Δ = 0.5), Meaningful effect (|d| > Δ = 0.5), Δ read back as you entered it, the raw conversion being the bounds note’s – or … not established (Δ = 0.5). TOST’s verdict reads the two-sided p as well: Statistically different but equivalent when both reject, Different, equivalence not established when only the two-sided does, and Inconclusive – neither difference nor equivalence established when neither does.
• p (equiv, adj) – under a p-value adjustment in addition mode, beside the p (adj) column that carries the adjusted two-sided p: the decision p after adjustment – it is the equivalence p, not the two-sided one, that enters the run-wide family, since it carries the decision – p (MET, adj) for the minimal-effect test; in replacement mode the decision column shows the adjusted value itself. The verdict follows the adjusted p.

The pairwise tables carry the same columns when an equivalence direction is applied to a pairwise expansion, the bounds note listing a raw Δ per comparison where they differ.

Categorical tests (chi-square, Fisher’s exact, etc.) #

The categorical tests print their results row like any other – the per-group N, the statistic, df, p, the effect size and the verdict, Significant association or No significant association – and, with Joint frequency table on, the table of counts the test ran on beneath it. Fisher’s exact test has no statistic column: its p comes straight from the hypergeometric distribution, its effect size is Cramér’s V or the odds ratio – on a 2×2 table with the exact interval fisher.test reports – and under a one-tailed direction, offered on a 2×2 table alone, a note under the table names the odds the alternative ordered. A table with three or more rows and columns above 1,000 cases, any table above 30 cells, or one whose enumeration exhausts R’s workspace, takes a seeded Monte-Carlo p over 10,000 replicates and a note says so, flagging a p at the simulation’s floor of 1/10,001 as an upper bound. McNemar’s test switches to the exact binomial test on the discordant pairs below 25 of them, with a note giving the count, and reports NA with a note when there are none; a Stuart-Maxwell matrix that is fully singular – empty or perfectly redundant categories – reports NA with a note, a partially singular one is inverted on its effective rank, with the df reduced to match and a note calling the result approximate, and a 2×2 table hands the test to McNemar’s, continuity correction included. Cochran’s Q and the Cochran-Armitage trend test count the higher of a binary outcome’s two values as the success, and a note names it, so the frequency table – and the sign of the trend – reads correctly.

• Contingency table – Contingency table: {variable}, one per dependent variable: the observed count in each cell with its column percentage, the row and column totals and, for the chi-square and goodness-of-fit tests, the expected count (Exp:) and the standardized residual (Std. res:) beneath each count, a note under the table explaining the two prefixes. A residual in bold clears the critical |z| Bonferroni-adjusted over the number of cells – the note gives the value and the cell count – which marks a cell extreme enough to survive scanning the whole table rather than one past a rule-of-thumb ±2, and the category proportion plot marks the same cells by the same rule. The paired tests’ table crosses the two conditions of one variable – the row header names the first, a spanning header the second – so the off-diagonal cells are the discordant pairs. A numeric dependent variable with 20 or fewer distinct values gets the same table of counts, without expectations.
• Value – the category of the dependent variable each row counts; on a paired table, its category at the first condition, whose name replaces this header.
• Count – the goodness-of-fit table’s one column: the observed count in each category with its percentage and, beneath it, the expected count and standardized residual under your expected distribution.
• Observed vs expected – Observed vs expected: {variable}, the goodness-of-fit test’s table, shown whether or not frequency tables are on, since it is the test’s core output: one row per category of the one variable, no cross-tab – see goodness-of-fit test. The results row carries the χ², its df – the number of categories less one – the p and, with effect sizes on, Cohen’s w; where an expected count falls below 5 the p is recomputed by seeded Monte-Carlo simulation over 10,000 replicates and a note gives the minimum expected count and the replicate count – the χ² and df stay the asymptotic ones. With visualization on, an observed-vs-expected chart accompanies the table.
• Stratum: {variable} = {level} – the Cochran-Mantel-Haenszel test’s per-stratum tables, one each after the pooled Contingency table – the slices the test pools over, so you can see whether the association points the same way in each – see stratum. Its results row carries the Mantel-Haenszel χ² with df = (I − 1)(J − 1) and, on a 2×2×K table with effect sizes on, the common odds ratio with its interval, the note beneath naming which odds it divides by which; a Breslow-Day (Tarone) line reports whether the strata’s odds ratios are homogeneous, at the assumption test significance level – when they are not, the common ratio may mislead and the per-stratum tables are the report – see Breslow-Day test. Further notes say when the pooled table was too sparse for the χ² approximation and the exact conditional test ran instead – a p with no χ² – when strata hold a zero cell, when strata of one observation were left out, and when Breslow-Day could not be computed at all.
• Cohen’s g by category pair – Cohen’s g by category pair: {variable}, the Stuart-Maxwell test’s effect-size table: one row per discordant category pair, each a genuine 2×2 Cohen’s g with its interval and magnitude, since a single g averaged over three or more categories is not one.
• Category pair – the two categories the row’s g is computed between, A vs B.

Correspondence analysis results #

Correspondence analysis. The section Include correspondence analysis adds to a chi-square or Fisher card: one block per dependent variable, headed by its name, with the inertia table, a coordinate table per margin, each headed Categories of {variable} – and the correspondence map, a note under each table saying how to read it. See correspondence analysis for what the decomposition is.

Multiple correspondence analysis. The one run-level section All selected variables together draws instead: the inertia table, adjusted for the indicator coding, a line counting the variables, categories and complete cases, a note naming any variable left out for taking one value and any pair of variables that split the cases identically, and one coordinate table over every category, with a Variable column naming each category’s. See multiple correspondence analysis.

• Categories of {variable} – the coordinate table of one margin, one row per category of that variable.
• Dimension – in the inertia table, the dimension’s number, every dimension the table has – one fewer than its smaller side; in the coordinate tables, Dimension 1 and Dimension 2, the category’s coordinates on the two dimensions the map draws, the only ones printed there.
• Inertia – in the inertia table, the dimension’s share of the total association in absolute terms – summed over the dimensions, the table’s χ² without the continuity correction over n, which the line under the table quotes; in the coordinate tables, the category’s share of that total as a percentage. See inertia.
• % of inertia – the dimension’s share as a percentage of the total; % of adjusted inertia on the MCA card, after Greenacre’s adjustment for the indicator coding, which is why those percentages do not sum to 100 and are not comparable with a two-way card’s.
• Cumulative – the running total, so the two dimensions on the map can be read off as one figure.
• Category – the category the row places, of the variable the table is headed by.
• Mass – the category’s share of the table’s count – see mass: a rare category moves the solution little however far out it lies.
• Contribution {n} – how much of that dimension the category built, as a percentage summing to 100 down the column – the few largest name the dimension – see contribution.
• Quality {n} – how well that dimension represents the category, the squared cosine, and Total quality the same over the drawn dimensions together; a category with low total quality is not described by the map however far from the origin it sits, and contribution and quality answer opposite questions – see quality.

A table the decomposition cannot place is refused with the reason in place of the tables: fewer than 2 rows or 2 columns left after missing values are dropped, or a table whose every cell matches independence exactly; the joint analysis also refuses fewer than three variables taking more than one value, fewer complete cases than variables, and a solution in which no dimension carries more than one variable’s worth of the coding – the variables look mutually unrelated.

Pairwise comparisons #

Pairwise comparisons. The card holds the flat tables of pairs – one per dependent variable, headed Pairwise comparisons: {variable} – that two kinds of run produce: a two-sample test on a factor with three or more levels, which is expanded over every pair and prints no omnibus row – see choosing a test – and the breakdown Include post-hoc tests adds to the omnibus tests that offer no method select – the chi-square test of independence, Fisher’s exact test, Anderson-Darling, Bartlett’s, Fligner-Killeen and the mutual information test – each pair being the omnibus test run on that pair’s own sub-table or sub-sample. Both are laid out per Pairwise comparison format, as a matrix or as a long table, and the Post-hoc for: {variable} tables the method-driven post-hocs print – Tukey, Dunn’s, Conover’s and the rest – share the layouts and most of the columns, with the differences under post-hoc tables. See pairwise comparisons.

Pairwise analysis summary. The box above the tables counts the variables analysed and their comparisons in total and, under a p-value adjustment, names the method and the family it was applied over: within each variable’s own family of {n} comparison(s) for an omnibus test’s opt-in breakdown, pooled with every other test in this analysis for an automatic expansion. The post-hoc tables carry their own Method: and adjustment lines; this box is what keeps the flat tables’ correction explicit.

• Matrix format – the levels on both axes and the lower triangle filled, every cell read as row − column: the mirrored orientation is derived from the fitted one – a signed statistic negated, a variance F reciprocated, Mann-Whitney’s U, the signed-rank V and the sign test’s count complemented against their maximum, a symmetric statistic (χ², D, T.AD, I, JSD) kept, the difference and its interval negated and swapped, the effect size flipped on its own scale, Glass’ Δ₁ and Δ₂ trading places, and a one-sided equivalence p replaced by the second fit the run made against the mirrored hypothesis. A cell stacks up to seven lines: the statistic with its df and significance stars; the estimated difference, where the pair’s test reports one, and the difference’s interval; p – p (two-sided) under an equivalence direction – and p(adj) in addition mode; the equivalence decision p, p(equiv), p(NI), p(sup) or p(MET), with its adjusted twin; the effect size with its interval and SE; Δ₂ for Glass’ Δ. Cells are shaded by the decision p – the adjusted one whenever an adjustment ran, the equivalence p under an equivalence direction – a failed pair prints Error, and the legend beneath keys the glyphs.
• Long format – one row per pair, with the columns below in this order: the comparison, the two groups’ statistics, the difference and its interval, the statistic, df, the p-values, the effect size with its interval and SE, and the verdict.
• Comparison – the pair the row compares, {group1} vs {group2}, the first level minus the second wherever a sign appears; on a post-hoc table it is written {group1} – {group2}.
• Group 1 and Group 2 – the two members of the pair, each cell stacking that group’s summary statistics as you enabled them – n, M, Mdn, Mode, SD, SE, the interval and, in a design with a within-subjects factor, the within-subject interval, Q1, Q3, IQR, Min, Max, Mean rank – so a pair is read against its own descriptives without leaving the row; the two columns are absent when no summary statistic is on.

The Difference column and its interval are the pair’s own estimate – present for the location tests, which report one, and absent for the dispersion, whole-distribution and categorical pairs, which do not – read under Difference and CI of the difference; under an equivalence direction the interval is built at 1 − 2α, as the bounds note above the table says. The statistic column is headed by the test’s own symbol, as on the Overall test results table, with the stars of the row’s decision p and the signed-rank V and sign-test count printed as ratios; df is the pair’s – Student’s expansion’s pooled N − k, Welch’s and Yuen’s fractional, df (num, denom) for the F-test – see df. The p-values follow the one-factor table – p, p (adj) in addition mode, and under an equivalence direction p (two-sided), p (lower), p (upper), p (equiv) or p (MET) and their adjusted forms, read under equivalence test results. The effect-size column carries the measure’s symbol, its CI and, on request, its SE, with Δ₂ beside Δ₁ for Glass’ Δ – read under Effect sizes.

• Interpretation – the short verdict of a pair: Significant or Not significant at the significance level, read from the adjusted p where one was applied; Significantly greater or Significantly less under a one-tailed direction; Equivalent, Different but equivalent, Different, not equivalent or Inconclusive under TOST, and Non-inferior, Superior or Meaningful effect – or their not established forms – under the other equivalence directions; Undetermined where the p is missing, Failed where the pair errored. Then the effect’s magnitude against the measure’s benchmarks, and a direction line – Group 1 > Group 2 – read from the pair’s own estimate where it has one and from the family’s summary statistic otherwise, prefixed Direction confirmed: or Unexpected direction: on a significant one-tailed pair. On a post-hoc table the verdict is the decision alone. See interpretation.

Notes under the tables say what the columns cannot: that the intervals are not adjusted for multiplicity when the p-values are; that the within-subject interval is the Cousineau-Morey one; that a Hedges’ g interval is the uncorrected d’s; the trim level of a trimmed-means expansion; that a pairwise JSD is in bits against a ceiling of 1 bit; and, per pair, the diagnostics the one-factor table prints – expected counts below 5, a Monte-Carlo or exact branch, ties dropped, a continuity correction that had no effect, a median interval whose coverage is not the requested level. Student’s expansion adds one more: every pair was tested against the within-group error pooled over every group, on N − k degrees of freedom rather than the two groups’ n₁ + n₂ − 2, and the note names Welch’s t-test as the way to a per-pair separate-variance error – the pooling follows from the test’s equal-variance premise, so the test selector is the control over it.

Pairwise contingency tables. When the pairs come from a categorical test and Joint frequency table is on, a collapsible Pairwise contingency tables: {variable} panel follows the table, holding the sub-table each pair was tested on, headed A vs B – the counts behind the pair’s χ², Cramér’s V or odds ratio; Cochran’s Q’s pairwise McNemar tests get the same panel, each pair’s 2×2 table of agreements and discordances. A Stuart-Maxwell expansion adds Cohen’s g by category pair: {variable}, one g table per pair, and a Kolmogorov-Smirnov expansion the per-pair where the distributions separate panel.

Post-hoc tables #

A method-driven post-hoc – Include post-hoc tests on a one-factor omnibus with a method select – prints one table per dependent variable in the Post-hoc comparisons card, headed Post-hoc for: {variable}, in the layout chosen under Pairwise comparison format; the many-to-one methods always render long. Up to four lines sit between the heading and the table: Method: naming the method as the select spells it; the adjustment line – P-values adjusted using {method}, which for a self-correcting method names its internal control (Tukey’s studentized range (exact) or Tukey HSD, multivariate t (exact), Dunnett’s multivariate t (exact), Studentized maximum modulus), or P-values not adjusted – with a caveat under Holm, Hommel, Hochberg, Benjamini-Hochberg and Benjamini-Yekutieli that the intervals come from a single-step correction, so a row can be significant while its interval includes zero; the control a many-to-one method actually contrasted against, when the chosen one had no usable data; and Reference df = {df} for Conover’s test, whose one (n − 1)(k − 1) degrees of freedom every pair shares. A warning above the table flags an omnibus that was not significant, since a grid of significant pairs under a non-significant omnibus is the classic protected-test misread.

• Difference – on a post-hoc table, the contrast’s estimate, the first level minus the second: the difference of the two levels’ means under Tukey, Games-Howell, Dunnett’s tests and the pooled pairwise t-tests, of their estimated marginal or adjusted means under the emmeans-driven methods, and of their trimmed means under lincon; the rank methods – Dunn’s, Conover’s, Nemenyi – and the divergence pairs report a statistic and no difference, so the column is absent. Its interval is headed simultaneous CI where the method widened it over the family – see {level}% simultaneous CI.

The statistic column is the method’s own – Z for Dunn’s tests, t for Conover’s, Games-Howell, Dunnett’s, Dunnett’s T3 and every emmeans contrast, q for Nemenyi, JSD for the divergence pairs, none for Tukey and the pooled pairwise t-tests, whose rows carry the difference, its interval and the p alone – with the stars of the row’s p, which is already the adjusted one; a df column appears where the method carries one per pair – the emmeans contrasts’ residual df, Games-Howell’s and Dunnett’s T3’s Welch df – and not for Tukey, Nemenyi, Dunn’s or the one-way Dunnett. In the matrix layout each cell stacks the statistic, its df, Diff, the interval and p, the levels’ summary statistics sit under the column headers, and the legend names the lines. On a design with a within-subjects factor a note above the block names the error term the contrasts were computed against: the repeated measures and mixed families give each comparison its own – the error of the cells it involves, not the pooled term behind the omnibus – so a contrast stays valid when sphericity fails, at the cost of fewer degrees of freedom and a verdict that need not agree with the omnibus; the ART families pool, and say so – see error term. A factorial or mixed design lays its contrasts out per effect instead, under factorial ANOVA.

Classification (ROC) results #

Classification (ROC) analysis. The section Include classification (ROC) analysis adds after the pairwise comparisons: one table with a row per dependent variable – per group pair on a pairwise expansion, and two rows per fit under the cost-weighted rule – with a legend beneath, the notes the run calls for, and, when AUCs are compared, the AUC comparison table. A curve that could not be fitted – a pair left with fewer than two groups, or fewer than four scored cases – is listed in an error summary under the table, not as a row. What the numbers mean is the regression page’s: ROC curve, AUC and the metrics after it; the curve itself is drawn by the ROC curve plot.

• Variable – the dependent variable whose scores are thresholded.
• Comparison – the group pair the row’s curve was fitted on, A vs B, present only on a pairwise expansion.
• Predicted group – the group higher scores signal, the one a case at or above the threshold is classed as: the level the fit left as the cases, chosen so that the AUC is at least 0.5 – except under Compare AUCs, where one orientation is held across the variables and a variable that discriminates in reverse keeps this label with an AUC below 0.5, as the legend then says.
• Other group – the paired group, the one lower scores signal – the other of the N column’s count.
• AUC – the area under the curve: the probability that a random member of the predicted group outscores a random member of the other, 0.5 chance and 1 perfect separation – see AUC for the conventional bands and why they are guidelines, not laws.
• {level}% CI – the AUC’s interval, DeLong’s closed form or the bootstrap per AUC confidence interval; the legend names the method.
• Brier – the mean squared error of the Platt-calibrated probabilities against group membership, 0 perfect and lower better: where the AUC asks whether the scores rank the groups correctly, the Brier score asks whether the probabilities they imply are right – see Brier score. The calibration is fitted on the same cases, so it summarises fit rather than forecast skill, and the cell is blank where every score is the same.
• Brier skill – the same score rescaled against always predicting the base rate, comparable across variables with different group balance: 1 perfect, 0 no better than the base rate, negative worse – see Brier skill.
• N – the cases the curve was fitted on, total (predicted/other); under Compare AUCs the subset every variable shares, as a note beneath says.
• Worse to misclassify – under the cost-weighted rule alone, which of the fit’s two rows this is: Missing a positive – the cost ratio weighing a missed case of the predicted group as the costlier error – or False alarm, its reciprocal.
• Threshold – the cutoff the rule chose, with its bootstrap interval in brackets: scores at or above it are classed as the predicted group – prefixed ≥, or ≤ for a reverse-discriminating variable under Compare AUCs, whose operating point is taken from the mirrored rule. Shown, with the five metrics after it, while Classification metrics at optimal threshold is on.
• Sensitivity – the share of the predicted group’s cases the threshold catches, the true positive rate – see sensitivity; it and specificity trade off as the threshold moves, and neither depends on the prevalence.
• Specificity – the share of the other group’s cases the threshold rules out, the true negative rate – see specificity.
• PPV – the share of the cases classed as the predicted group that belong to it – see positive predictive value; with NPV and accuracy it depends on the prevalence, so a sample whose group balance is not the population’s reads it at the population prevalence you enter, and the note under the table says which prevalence reached it.
• NPV – the share of the cases classed as the other group that belong to it – see negative predictive value.
• Accuracy – the share of all cases classed correctly, at the same prevalence as PPV and NPV.

The bracketed interval beside the threshold and each metric is a percentile bootstrap that re-chooses the cutoff in every resample, over the Bootstrap replications setting, so it prices the optimism of a threshold chosen on the cases it is scored on; the legend says so. Below the legend: a note that Compare AUCs dropped every case missing on any selected variable before fitting, when it did; the threshold rule – Threshold optimized by Youden’s J, … by closest point to (0, 1), or the cost-weighted line with the cost ratio and the prevalence it used, population prevalence 0.3 where it applies and the sample prevalence elsewhere when a pairwise expansion mixes the two; and, when a prevalence was entered, which group it was tied to – or that it reached nothing here, because the level named is not a group of this comparison or no group was named, in which case the sample prevalence stands and the note says so, so an entered prevalence is never silently assumed to have applied.

AUC comparison (DeLong’s test) #

AUC comparison (DeLong’s test). With Compare AUCs on and two or more curves fitted, every pair of dependent variables is tested with DeLong’s test for correlated AUCs – see DeLong’s test – in one table, or on a pairwise expansion one section per group pair headed A vs B, each with its cross-variable comparisons. A note beneath restates the columns, and the adjustment line names the method and its family: every AUC comparison in the analysis – all variable pairs, group pairs and batch iterations – not only the rows on screen.

• Variable 1 and Variable 2 – the two dependent variables whose AUCs the row compares.
• Δ AUC – AUC(Variable 1) − AUC(Variable 2).
• Z – DeLong’s statistic for the difference between two AUCs on the same subjects, with the stars of the row’s decision p – see z.

The p column follows it, and p (adj) in addition mode – see p (adj); in replacement mode the adjusted value stands in the p column, which is why the line beneath names the method. A pair whose test failed is listed in an error summary under the table.

Factorial ANOVA #

Factorial ANOVA results. The card – Factorial ANCOVA results when covariates are assigned – holds one block per dependent variable, headed Factorial ANOVA: {variable} or Factorial ANCOVA: {variable}, for the Factorial ANOVA test: two or more grouping variables in one model, a run with one being refused (when a test refuses to run) rather than collapsed to a one-way ANOVA. Covariates join the design effects in the same source table and make the adjusted means covariate-adjusted; everything else reads the same. Each block runs, in order: the source table, the diagnostics, the post-hoc contrasts, the adjusted means, the heterogeneous-slopes follow-up and the descriptives. See factor, cell and full crossing for the design’s vocabulary.

The source table has one row per effect of the full crossing – every main effect and every interaction – plus a row per covariate, and closes with the Error and Total (corrected) rows that complete the partition, so every number above can be checked against it; a note beneath names the type of sums of squares, and a non-partial effect size gets a second note on what it divides by.

• Source – the effect the row tests: a factor, an interaction written with ×, or a covariate. With covariates present each factor row is tagged (factor) and each covariate row (covariate), so the two kinds of predictor are told apart at a glance; an interaction row carries no tag, since A × B already reads as a design term.
• SS – the sum of squares the chosen type credited to that source – see sum of squares. Under Type III each term’s is its unique contribution, so on an unbalanced design the effect rows do not add up to the corrected total.
• df – the degrees of freedom of the row’s statistic: on a source table the effect’s and the error’s in one cell, 2, 189, the pair the F is referred to, the closing rows carrying their own alone; on a contrast or pairwise row a single value – the residual df of an emmeans contrast, a Kenward-Roger estimate on the ART cards, Student’s expansion’s pooled N − k, Welch’s and Yuen’s fractional – see df.
• MS – the mean square, SS over df – see MS; the Error row’s is the denominator of every F above it, and the corrected total deliberately shows none, since SS/df there is the sample variance, not a source-table term.

Then F – the row’s mean square over the error mean square, with the significance stars of its decision p – see F; p, and p (adj) in addition mode – every effect row enters the run-wide pool, the covariate rows excepted; the effect size the Effect size method resolved to – partial η² by default, or η², ω² or ε² – headed by its symbol, with its interval under Effect size confidence intervals, partial η² standing in with a diagnostic when the package could not compute the chosen one; and the Interpretation verdict – Significant main effect or No main effect, Significant interaction or No interaction, Significant covariate effect or No covariate effect, Undetermined where the p is missing – read from the adjusted p where one was applied, followed by the effect’s magnitude against the measure’s benchmarks.

η², ω² and ε² are each the effect’s share of the corrected total sum of squares – the row at the foot of the table – which is what makes them checkable by hand, and the note under the table says so, naming the measure you chose: on an unbalanced design under Type III the shares do not add up to the model’s R², and a package dividing by the effect’s own SS plus the error SS reports slightly different values. Partial η² is unaffected.

With post-hoc tests on, the Post-hoc comparisons section follows – read under Post-hoc comparisons – with one block per effect term: Main effect: {factor}, the pairwise contrasts of that factor’s levels marginalised over the other factors – every level against the control under Dunnett’s test – and Interaction: {factors}, every cell-pair contrast, followed by two Simple effects: {factor} comparisons at each level of {other} tables, one factor’s contrasts within each level of the other, with that level in the table’s first column – see simple effect. Three-way and higher interactions keep their omnibus row and get no contrasts, and the diagnostics say so; a covariate has no levels to contrast. Each block opens with its adjustment line – the interaction cell-pairs take the exact multivariate t under every self-correcting method, as Post-hoc method says – and a warning where the effect’s omnibus p was not significant, and its rows carry the Contrast, the Difference on the estimated marginal means, its SE, df, t with stars, the adjusted p and the interval.

An Adjusted means (estimated marginal means) section always follows – whether or not post-hoc tests are on, since the marginal means are what the source table’s nulls are about – read under ANCOVA: Adjusted cell means (A × B) over the full crossing first, then Marginal means by factor, each with its SE and interval. Without covariates they are the equal-weighted marginal means a Type III main effect tests, which on an unbalanced design part from the sample-size-weighted raw means in the descriptives – see estimated marginal means; with covariates they are covariate-adjusted, and a line beneath names each covariate and the value it is held at, its mean over the fitted rows. With covariates present the Heterogeneous slopes (regions of significance) section follows, contrasting the between-cell crossing pair by pair.

The descriptives close the block – read under the mixed card: Descriptive statistics by {variable} (between-subjects), one table over the levels of every grouping factor, and Descriptive statistics by cell (A × B) over the full crossing, each with the summary statistics you enabled, computed over the rows the model was fitted on – so the cell counts make an imbalance visible, and the raw means sit under the adjusted ones.

Reading a factorial’s post-hoc? Decide from the source table which terms matter, then drill into that block: a significant interaction is read through the simple effects, which say where it lives, and only a non-significant one leaves the main effects as the primary read.

ANCOVA #

The ANCOVA results card holds one block per dependent variable, headed ANCOVA: {variable} – the one-way ANCOVA entry’s card; a factorial ANCOVA is read under factorial ANOVA. Each block runs, in order: the source table, the diagnostics, the adjusted means, the post-hoc contrasts, the heterogeneous-slopes follow-up and the descriptives – so the raw means sit under the adjusted ones, and how far the covariate moved each group is read off the pair. See ANCOVA for the design.

The source table is the factorial one: one row per covariate, tagged (covariate), then the grouping factor, tagged (factor), each with SS, df (effect, error), MS, F and p, the effect size the Effect size method resolved to – partial η² unless you picked η², ω² or ε² – with its interval, and the Interpretation verdict, which reads Significant covariate effect or No covariate effect on a covariate row; an Error row and a Total (corrected) row close the partition, a note names the type of sums of squares, and a non-partial measure gets the note on what it divides by. A Diagnostics box under the table names what the fit fell back on – an effect size the package could not compute, a post-hoc term that failed.

• Error – the row that closes the partition: the residual df, SS and MS – the denominator every F above was tested against – with the F, p and effect-size cells left blank.

• Total (corrected) – the mean-centred total over the fitted rows, df and SS alone; SS/df there would be the sample variance, not a source-table term, so its MS cell is empty on purpose.

• Adjusted means (estimated marginal means) – one row per group: its estimated marginal mean at the covariate values a line under the heading names – Covariates are held at their means: …, each covariate’s mean over the rows the model was fitted on – with its SE and interval, beside the unadjusted mean of the same rows. The same section heads the mixed and multivariate cards: there it appears when post-hoc tests are on or a covariate is in the model (the mixed families) or always (MANCOVA, Mixed MANOVA), holds the equal-weighted marginal means when no covariate is present, prints Adjusted cell means over the full crossing of two or more factors first and Marginal means by factor beneath, and on a card holding several dependent variables carries a Variable column naming each row’s.

• Group – the level of the grouping variable the row describes – the same column heads every per-group table of the card.

• Condition – the level of the condition variable the row describes, on the repeated-measures cards.

• Unadjusted mean – the group’s plain mean over the rows the model was fitted on, with the N and SD of those rows beside it; it parts from the adjusted mean by however far the group’s covariate values sit from the value the model holds them at.

• Adjusted mean – the group’s mean predicted at the covariate values named under the heading, in bold – see adjusted mean; the SE and interval beside it are the estimate’s own.

• Factor – the factor a row belongs to: in the marginal table of the mixed families, the one whose level the row’s mean is for; on a source table, the tag marking a design term where a covariate row is tagged (covariate).

• Level – the level of that factor the row’s mean is for.

• Post-hoc comparisons (adjusted means) – every pair of groups contrasted on the adjusted means (emmeans), one row per pair with the Contrast, Difference, SE, df, t, p and interval – headed simultaneous CI where the method widened it over the family – read as under Pairwise comparisons; a line above names how the p-values were adjusted, and Dunnett’s test says which control it contrasted against when the chosen one had no usable data. Appears when Include post-hoc tests is on.

• Contrast – the pair of levels the row compares, the first level minus the second, as every contrast table on the card names it – see contrast.

• {level}% simultaneous CI – the interval of the difference widened over the whole family, so that every contrast’s interval covers its true difference at once – see simultaneous confidence interval; a per-comparison interval is headed plain {level}% CI, and Pairwise comparisons says which methods give which.

A covariate is always present here, so the Heterogeneous slopes (regions of significance) section follows the contrasts – see heterogeneous slopes results.

• Descriptive statistics by group – the unadjusted per-group summaries over the rows the model was fitted on, one column per group and summary statistic you enabled; it sits under the adjusted means on purpose, so the raw means beside the adjusted ones show how much the covariate moved each group. The same table closes the MANOVA and MANCOVA cards with one row per dependent variable, and the repeated measures MANOVA card as Descriptive statistics by condition, with a collapsible Frequency table: {variable} per dependent variable when Joint frequency table is on.

Mixed ANOVA #

The Mixed ANOVA results card – and the Repeated measures factorial ANOVA results card, Repeated measures factorial ANCOVA results with a covariate, which shares its layout – holds one block per dependent variable, headed Mixed ANOVA: {variable} or Repeated measures factorial ANOVA: {variable} and opening with the line N = {n} complete subjects: the complete subjects the fit ran on, so a listwise drop is never silent. In order: the source table, the diagnostics, Mauchly’s test, the post-hoc contrasts, the adjusted means, the heterogeneous-slopes follow-up and the descriptives; ART-ANOVA reads the same way.

The source table has one row per effect of the full crossing – every main effect and every interaction – plus a row per covariate, each with SS, df (effect, error), MS, the effect’s own SS error and MS error, F, p, the effect size the Effect size method resolved to with its interval, and the Interpretation verdict – Significant main effect, No interaction, Significant covariate effect and so on, with the effect’s magnitude. On a within-subjects factor with three or more levels the ε (GG), p (GG), ε (HF) and p (HF) columns join it, filled on the within rows alone – the same quantities the assumption check prints – and a within row’s verdict follows the corrected p, tagged GG-corrected or HF-corrected: Greenhouse-Geisser when ε̂ (GG) is below 0.75, Huynh-Feldt from 0.75 up, whatever Mauchly’s test says. Under the multivariate route the within rows carry Pillai’s trace in a V column with its approximate F instead, their sums of squares stay blank, a note under the table says which rows took which route, and the diagnostics name any term that had to stay univariate. A note names the type of sums of squares.

• Effect – the model term the row tests – a factor, an interaction written with ×, or a covariate – tagged (between), (within), (between × within) or (covariate) for the side of the design it belongs to, which is the error stratum its F was tested against.
• SS error – the sum of squares of the error stratum the row’s effect is tested against: a between effect against the spread between subjects, a within effect or an interaction against the spread of the changes within them – so a mixed table carries its error terms per row rather than as one closing row.
• MS error – that error sum of squares over its degrees of freedom, the denominator of the row’s F.
• Mauchly’s test of sphericity – one row per within-subjects term with three or more levels: the Effect, W, p and a verdict, Sphericity violated at the assumption test significance level or Sphericity assumption met; the note beneath says the ε correction is applied to every within row regardless, so a met row never means an uncorrected p. See sphericity and Mauchly’s test.
• W – Mauchly’s statistic, 1 when the variances of the pairwise differences are equal and smaller the further they part.
• Post-hoc comparisons – the section Include post-hoc tests adds. On a one-factor test it is one table per dependent variable, headed Post-hoc for: {variable}, read under Pairwise comparisons. On the factorial, mixed and ART families it is one block per effect: Main effect: {factor}, the pairwise contrasts of that factor’s levels marginalised over the others; Interaction: {factors}, the cell-pair contrasts, followed on the parametric families by two Simple effects tables – one factor’s contrasts at each level of the other, the read that says where an interaction lives (see simple effect) – each block with its own line naming the adjustment, the columns of a post-hoc table, and a warning when the effect’s omnibus p was not significant, so a grid of significant pairs under a non-significant F is not read as a finding. Three-way and higher interactions keep their omnibus row and get no contrasts, and the diagnostics say so. A note above the blocks names the error term the contrasts share or do not. Mixed MANOVA titles the section Post-hoc for: {variable}, one per dependent variable.

The Adjusted means (estimated marginal means) section appears when post-hoc tests are on or a covariate is in the model – read under ANCOVA – and, with a covariate, Heterogeneous slopes (regions of significance) follows it: a mixed design contrasts on both sides, so its rows carry the Contrasts and At columns – see heterogeneous slopes results.

• Descriptive statistics by factor and by cell – one table over the between-subjects factors, headed Descriptive statistics by {variable} (between-subjects), one over the within-subjects factors, … (within-subjects), and with two or more factors one over the full crossing, Descriptive statistics by cell ({factors}) – by {between} × {within} on the Mixed MANOVA card – each with a column per level (factor=level where a table spans two factors) and summary statistic you enabled, computed over the complete subjects the F was, so the N beside a mean is the tested one; the cell counts make a between-cell imbalance visible, and a repeated level carries the within-subject interval as a second interval column. The factorial ANOVA card prints its between and cell tables the same way.

Sphericity and the two corrections? Sphericity is the equal-variance-of-differences assumption a repeated factor with three or more levels adds, and the two corrections shrink the F’s degrees of freedom when it fails – see sphericity, the Greenhouse–Geisser and Huynh–Feldt corrections. Report whichever the verdict’s tag names, with its ε.

ART-ANOVA #

The ART-ANOVA results card – Repeated measures factorial ART-ANOVA results for the within-only test – shares the mixed ANOVA layout: one block per dependent variable, headed ART-ANOVA: {variable}, with the N = {n} complete subjects line above the table, since the aligned-rank fit needs every subject in every within cell as the parametric one does. The source table has no sums of squares – the aligned-rank fit partitions none – no sphericity columns and no Mauchly’s table, since ranks sidestep the assumption; each row carries its Effect with its tag, df (effect, error – the error df is a Kenward-Roger estimate on the underlying mixed model, so it need not be a whole number), F, p, and partial η² with its interval, the one effect size the test reports, recovered from F and the degrees of freedom. The table closes with the spread of the cell standard deviations over the rows the fit used, because the fit assumes them equal and corrects no departure – where they differ, every F above is anti-conservative, the between-subjects rows included. A covariate assigned to an ART test is dropped and the Diagnostics box names it; the same box reports a fit that failed ARTool’s own alignment check – the aligned responses not summing to zero, or an F not near 0 on an aligned response not of interest, usually an empty between × within cell or unequal between-cell sizes – since the F-values are uninterpretable then; and a fit whose denominator df could not be estimated stops with an error naming the cause, too few subjects per between cell or collinear factors. See aligned rank transform.

With post-hoc tests on, the Post-hoc comparisons section holds one block per main effect and two-way interaction, computed by ARTool’s art.con, which re-aligns the ranks for each term: a main effect’s block is the pairwise contrasts of that factor’s levels on its own aligned-and-ranked response; an interaction’s is difference-of-differences contrasts, labelled (a₁ − a₂) × (b₁ − b₂) – how the gap between two levels of one factor shifts across two levels of the other – and there are no simple-effects tables, since a simple effect is not a valid contrast under alignment. Every pair in a block shares one standard error and one df, and the note under the block says so: read the pairs as a set against the spread of the cell standard deviations, since where the true cell variances differ one pair runs conservative while another runs liberal. The method dropdown chooses only the correction, as Post-hoc tests says, and the descriptives close the card as on the mixed one.

MANOVA / MANCOVA #

One card covers every dependent variable, titled by the test – MANOVA results, MANCOVA results, Mixed MANOVA results (Mixed MANCOVA results with a covariate), Repeated measures MANOVA results (Repeated measures MANCOVA results), Johansen’s heteroscedastic MANOVA results or Energy test of equal distributions results – and opens with a line naming the dependent variables, the grouping or condition variable and the covariates, then a warning with the counts when subjects were dropped for a missing within-subject cell (the repeated and mixed designs) or rows for a missing dependent-variable or covariate value (the independent ones). A run whose dependent-variable matrix is rank-deficient – two variables that are combinations of each other, one that reproduces a covariate, too few complete rows – stops with an error naming the rank it found and the variable count, and a MANCOVA whose complete rows leave fewer error degrees of freedom than dependent variables stops naming both. In order: the multivariate tests, the covariate tests, the univariate follow-ups, the discriminant analysis, the adjusted means, the post-hoc contrasts, the heterogeneous-slopes follow-up and the descriptives.

• Multivariate tests – Multivariate tests for {variable} on the MANCOVA card – the omnibus: four rows, Pillai, Wilks, Hotelling-Lawley and Roy, each with its value, approximate F, numerator and denominator df and p, all testing the same hypothesis; on a mixed design one set per effect, named in an Effect column with its tag. Under a p-value adjustment the adjusted p – one per effect, since one hypothesis is one adjustment – and its verdict print on Pillai’s row alone, and with effect sizes on that row also carries the multivariate partial η² (η²p, whichever univariate measure you picked) with its interval. Roy’s row is flagged in a tooltip and a note: its F is an upper bound, exact only when the effect spans one dimension, so its p is a lower bound – read Pillai’s when the four disagree, for that and for its robustness (see Pillai’s trace). Johansen’s test has one row, its own statistic with Johansen’s approximate F and fractional df; the energy test one row with E and a permutation p, no F and no df, R²ₑ as its effect size and a note giving the replicate count and the smallest p it can report.
• Test statistic – which of the four the row is: Pillai’s trace, Wilks’ Λ, Hotelling–Lawley trace or Roy’s largest root.
• Value – the statistic itself on its own scale: Pillai’s, Hotelling-Lawley’s and Roy’s grow with the separation, Wilks’ Λ falls from 1.
• Approx. F – the F the statistic is referred to – exact when the effect spans one dimension (two groups, or one dependent variable), where the four rows give one F, and an approximation otherwise.
• Num. df – the numerator degrees of freedom of that F.
• Den. df – its denominator degrees of freedom; fractional under Johansen’s test.
• Covariate multivariate tests (Pillai) – on the covariate-bearing cards, one Pillai test per covariate across the dependent variables, each Type III so the order of the covariates does not matter, with its approximate F, df and p; nuisance terms, so they stay out of the adjustment pool.
• Pillai – the covariate’s Pillai’s trace.
• Univariate follow-up tests – one row per dependent variable (per effect and dependent variable on a mixed design), of the kind matching the omnibus and named in the heading: One-way ANOVAs or One-way ANCOVAs, Mixed ANOVAs or Mixed ANCOVAs, Repeated-measures ANOVAs or ANCOVAs, ANCOVAs on the MANCOVA card, Welch’s ANOVAs after Johansen’s test – a pooled follow-up would re-import the equal-covariance assumption its omnibus drops – and energy tests after the energy test, with E and no df. Each row has the statistic, df and p, the sphericity columns on the within rows of the repeated and mixed designs, the effect size you picked with its interval, and the verdict, read from the corrected p and tagged as on the mixed card. The follow-ups are fitted on the rows the omnibus kept, are gated by it rather than pooled into the global adjustment, and are adjusted among themselves: within each multivariate effect the K per-variable p-values are corrected together by your adjustment method, and a note above the table records K and the method – see univariate follow-up tests.
• Dependent variable – the dependent variable the row tests on its own.
• E – the energy statistic of the row’s permutation test: the energy distance between the groups on the standardized variables, which grows with the sample and has no reference distribution but the permutation’s.

Descriptive discriminant analysis #

• Descriptive discriminant analysis – opt-in through Include descriptive discriminant analysis, MANOVA and MANCOVA only: the section names the combination of dependent variables the groups separate on – see descriptive discriminant analysis – in four tables, the functions, the standardized coefficients, the structure coefficients and the centroids, each with a note beneath on how to read it; under MANCOVA the functions come off the covariate-adjusted matrices. It sits beside the univariate follow-ups, not in place of them, since it tests no single variable.
• Function – the discriminant function’s number, in order of the separation it carries; the coefficient tables head their columns Function 1, Function 2, … – see discriminant function.
• Eigenvalue – the spread between the group means along the function over the spread within the groups.
• Canonical r – the canonical correlation between the function’s scores and group membership.
• % of variance – the function’s share of the total separation, its eigenvalue over the sum of them all.
• Wilks’ Λ – with the χ², df and p beside it, the sequential test of that function together with every function after it: the first row restates the omnibus, a later row asks whether any separation is left once the earlier functions are accounted for.
• Standardized canonical coefficients – the weight each dependent variable carries in each function once the others are held constant, in pooled within-group standard deviations – the table to read for what a function is built from; a variable can weigh heavily here while its own follow-up is not significant, which is exactly the separation the multivariate test found and the per-variable tests cannot see.
• Structure coefficients – the pooled within-group correlations between each dependent variable and each function; they describe one variable at a time and rank the variables much as the follow-ups do, which is why they are shown second.
• Group centroids – each group’s mean score on each function, in pooled within-group standard deviations, so a gap of 1 is one within-group SD of separation; under MANCOVA they are the adjusted means read along the functions.

The Adjusted means (estimated marginal means) section – read under ANCOVA – prints one row set per dependent variable, with a Variable column: MANCOVA’s covariate-adjusted group means, always; Mixed MANOVA’s marginal means per between-subjects factor and its between cells, always, covariate-adjusted when covariates are present; repeated measures MANCOVA’s condition means. The line beneath reads held at their means over each dependent variable’s complete cases when the fitted rows differ between dependent variables. MANOVA prints none, since its adjusted means would be the group means the descriptives already show.

With Include post-hoc tests on, each dependent variable gets a Post-hoc for: {variable} block after the adjusted means. MANOVA, MANCOVA, Johansen’s test and repeated measures MANOVA print a flat table – pairwise group contrasts, covariate-adjusted on MANCOVA, the post-hoc method you chose on Johansen’s Welch follow-ups, pairwise condition contrasts on the repeated design – read under Pairwise comparisons, with a Method: line naming the contrast family. Mixed MANOVA prints the per-effect blocks of the post-hoc section: the terms carrying a within-subject factor are contrasted on that variable’s own split-plot fit, so they match its univariate F, and each “omnibus was not significant” warning reads that variable’s follow-up p rather than the multivariate omnibus, since the multivariate test gates the whole set and not one variable’s contrasts. The energy test has no post-hoc block – there is no pooled term to contrast – and its follow-ups are the per-variable tests above.

The descriptives close the card: Descriptive statistics by group (MANOVA, MANCOVA, Johansen’s test, the energy test) or by condition (the repeated design), one row per dependent variable – read under ANCOVA – and on Mixed MANOVA the three per-factor tables of the mixed card.

Heterogeneous slopes results #

Every card fitted with a covariate closes with this follow-up – ANCOVA, factorial ANCOVA, mixed ANOVA, repeated measures ANOVA and its factorial variant, MANCOVA, repeated measures MANOVA and Mixed MANOVA. The model above holds one slope per covariate for every level, so each adjusted difference is read at that shared slope; where a pair’s slopes genuinely differ the difference between those levels is a function of the covariate rather than one number, and this section is where that function is reported – a note under the heading says the same, and configuration says when the follow-up appears. On the multivariate cards, where one card holds several dependent variables, each region is univariate and tagged with its dependent variable, since a multivariate region of significance is not a standard construct – a note above the tables says so. See heterogeneous slopes.

One table per covariate, under Slopes on {covariate}, with one row per contrast – the pair of levels compared, named in a Contrast column, and a Variable column naming the dependent variable where the card holds several:

• Contrasts – the two mixed families only (mixed ANOVA, Mixed MANOVA): whether the row’s pair is a Between-subjects or a Within-subjects one. A mixed design has two sides to contrast, so a row that did not name its side would be ambiguous.
• At – with it, the cell of the other crossing the pair was fitted in: the between cell for a condition pair, the condition for a group pair.
• Slope difference – the covariate × factor term the fitted model deliberately leaves out, estimated for this pair, with its SE, t and p: the pair-level version of the omnibus homogeneity-of-regression-slopes check, which can only say that some pair’s slopes differ, never which.
• Region boundaries – the Johnson-Neyman covariate values at which the pair crosses into or out of significance, at the confidence level. Only boundaries inside the observed range of the covariate are listed – outside it a crossing is extrapolation from the fitted line, not something the data speaks to – so a pair can show two, one or none; with none, the share beside it says whether the pair differs everywhere in range or nowhere.
• Cases in significant region – the share of the observations whose covariate value lies where the pair differs, counted over the actual data. A region covering 3% of the sample is a different finding from one covering 80%, and the slope test alone cannot tell them apart.

Beneath the table a Region of significance: {contrast} plot is drawn for each contrast whose own slope test rejects at the assumption test significance level, or for every contrast when Show every region of significance is on: the contrast’s difference against the covariate with its confidence band, green shading over the covariate values at which the two levels differ significantly, dashed lines at the Johnson-Neyman boundaries – see the Johnson–Neyman region.

A replacement, not a supplement. When the slope check fails, the assumption panel demotes the test for exactly this reason – one adjusted difference cannot hold across the covariate – and the region is what describes the groups away from the value the covariate is held at; the adjusted means above still describe them there. See heterogeneous slopes.

Batch analysis #

In batch mode every grouping variable (independent designs) or condition variable (dependent designs) gets its own card, titled Batch analysis {current}/{total}: {test} ({factor}) and listed in the contents as Comparison ({group}), each read as the one-factor card of that test; p-values pool across every card – see P-value adjustment for what enters the pool.

Pseudo-factorial analysis #

Under the combined strategy the card is titled Pseudo-factorial analysis: {test} ({factor}), listed in the contents as Comparison ({group}), and reads as the one-factor card of that test with the cells of the crossing as its groups.

P-value adjustment #

Every test in a run is one more chance of a false positive, so the run’s p-values are adjusted together under the global adjustment method – see multiple comparison adjustment for what the methods promise and which adjustment to use for the choice. The adjusted p appears beside the raw one or in its place as the display setting says, and the pairwise and post-hoc cards name the method and the family it ran over.

One pool holds every omnibus p-value of the run, one slot per hypothesis: one per dependent variable for the standard tests; every pair of a two-sample test expanded over three or more levels; every effect row of a factorial, mixed or repeated measures ANOVA and of ANCOVA except the covariate rows, a within-subjects row contributing the ε-corrected p its verdict reads; and one p per multivariate effect, written back to all four multivariate statistics, which test one hypothesis. A batch or pseudo-factorial run pools every iteration, as Analyze each variable separately does.

Three families are corrected apart from that pool, each under the same method: the post-hoc tables – unless the method carries its own control – and the opt-in pairwise breakdown of a chi-square or Fisher’s exact test, per dependent variable, so switching a breakdown on does not tighten the correction on every other test; the univariate follow-ups of a multivariate test, together within each multivariate effect – see MANOVA / MANCOVA; and the AUC comparisons, across every variable pair, group pair and batch iteration. A family of one comparison is left raw, and a covariate’s own p is reported unadjusted in its effects table.

With None selected in the settings a warning appears when the run is about to produce a family nothing else corrects: every batch and pseudo-factorial run, and a single run with several dependent variables under a test that fits them separately, with two or more factors, with a two-sample test expanding over three or more levels, or with post-hoc tests on under a method that takes the global adjustment. The exact multivariate t, Tukey HSD, Games-Howell, Nemenyi, Dunnett’s test and Dunnett’s T3 correct their own tables, so a one-way ANOVA under Tukey runs silent, as does a two-group t-test.

Significance follows the adjusted p. The Significant verdicts and the stars on a statistic read the adjusted p in both display modes, so in addition mode a statistic’s stars can disagree with the raw p column beside it – see p (adj).

How many tests am I running? Five dependent variables under one factor are five tests; a two-factor ANOVA adds three effect rows per variable, a four-group pairwise expansion six pairs per variable – see many tests at once.

Missing data #

Missing values are handled by the global missing data setting before the run: pairwise deletion – the default – gives each test the cases complete on its own variables, listwise deletion the cases complete on every selected variable, and imputation fills the blanks first. Listwise deletion is the usual choice in published comparisons, since every group comparison then runs on the same cases; the Sample size summary column shows what each test ran on.

Paired and repeated-measures tests need complete subjects. A within-subjects test – the paired t-test and Wilcoxon signed-rank test, repeated measures ANOVA and Friedman, Page’s trend test, McNemar, Stuart-Maxwell and Cochran’s Q – drops every subject missing a condition of the variable under test before fitting and counts them in a warning; the per-condition descriptives are computed on the same subjects, and a variable with no complete subject reports that instead of running – see complete subjects and the groups preview.

Visualization #

Include visualization adds a card of plots under the results – Distribution comparison for the numeric families, Distribution in one-sample mode – with the checked plots for each numeric dependent variable, split by group or condition in the order the result tables use – numerically when every level is a number, so that dose 1, 2, 5, 10 reads in order, alphabetically otherwise. A native factorial test with several factors, the combined-factor strategy and every mixed design split the box, violin, ECDF and mean and error bar plots on the full cell crossing, the x-axis naming the crossed variables (Gender × Treatment); in one-sample mode the same four plots draw a dashed reference line at μ₀ and the value axis expands to keep it in view. A group with no valid values for a variable is dropped from that variable’s plots and named in an Excluded (no data) note beneath, and a group of one is dropped from the violin and mean and error bar plots the same way; the corner of each distribution plot counts the cases every group’s figure was drawn from. Every plot can be resized and exported as SVG, PNG or JPG. The categorical tests get their own cards instead – Category proportions and Observed vs expected – the discriminant function plot joins this card, and the correspondence map is a card of its own.

Box plot #

• Box plot – on by default: one grouped box plot per dependent variable – the box from Q1 to Q3, the median line, whiskers to the furthest values within 1.5 IQR and a diamond per value beyond them – see box plot. Hovering a box lists its n, the five numbers, the IQR, the mean when shown and the outliers.
• Show outliers – on by default: a diamond at each value beyond the whiskers – see Tukey’s fences; off, the whiskers still stop at the fences and the values beyond them are not drawn.
• Show mean – on by default: a hollow circle at each group’s mean, so a mean pulled away from the median shows the skew – see mean.
• Show notch – off by default: cuts a notch into each box spanning the median’s interval at the confidence level, interpolated from the order statistics and so asymmetric on skewed data – see interval for a median; two notches that do not overlap suggest medians that differ. A group too small for the interval gets no notch, and an interval wider than its box is clamped to the box’s edges, each named in a note beneath the plot.
• Show data points – off by default: every observation jittered beside its box, the outliers among them drawn as diamonds while Show outliers is on.

Violin plot #

• Violin plot – off by default: each group’s distribution as a kernel density outline, wide where values are dense, which shows the second peak or the long tail a box hides – see violin plot for what the smoothing costs. A group with fewer than two values or no spread has no density and is dropped with a note beneath.
• Show inner box plot – on by default: a miniature box plot inside each violin – a white dot at the median, a bar from Q1 to Q3 and a line out to the whiskers.

ECDF plot #

• ECDF plot – off by default: each group’s empirical cumulative distribution function – the share of its values at or below each value, a step curve rising from 0% to 100% – see ECDF and two ECDFs together; two groups with the same mean and SD can still part here, at a floor, a ceiling or a cluster. Under the Kolmogorov-Smirnov test on two groups the plot marks the largest vertical gap between the curves at the value it falls at, labelled D⁺ or D⁻ – see D⁺ and D⁻.
• Show median reference line – on by default: a dashed horizontal line at 50% and, from it, a vertical drop to the axis at each group’s median.
• Confidence band: – the shaded band around each curve at the confidence level, pointwise or simultaneous – see Wilson band and DKW band for the difference and when each is the one.
• Wilson (pointwise) – the default: each point of the curve gets the Wilson interval of a proportion, so the band narrows to nothing at the ends where the curve is pinned at 0% and 100%; read it one value at a time – see Wilson band.
• DKW (simultaneous) – a band of constant thickness that covers the whole curve at once (Dvoretzky-Kiefer-Wolfowitz), the honest choice when the curves are scanned along their length rather than read at one value – see DKW band.
• None – no band.

Mean and error bar plot #

• Mean and error bar plot – off by default: a dot at each group’s mean with whiskers spanning what Error bars show selects – the confidence interval of the mean by default, the standard error or the standard deviation – and a caption naming which, with the level for an interval. In a design with a within-subjects factor – dependent, or mixed on its between × within cells – the whiskers are the Cousineau-Morey within-subject interval or its difference-adjusted form, the caption says how to read them, and subjects missing a cell are dropped from the estimate and counted beneath. Hovering a group lists its mean, SD, SE, the whisker’s bounds and n.

Paired lines plot #

• Paired lines plot – off by default; offered for dependent and mixed designs: one faint line per subject through its values across the conditions, aligned by the Subject ID, with a bold mean line over them – see repeated measures. Lines that slope the same way are a consistent effect, crossing lines an effect that varies by subject, and the test says whether the mean line’s trend is more than chance. With two or more condition variables fitted together each x-position is one cell of their crossing, so a line spans every combination. A subject’s repeated rows in a condition are averaged and a subject missing a condition is left out, as the test does with them, and a toast counts each; the corner n is the subjects drawn.
• Show mean line – on by default: the bold line through the condition means, each labelled M = …, with a legend telling it from the subject lines.

Interaction plot #

• Interaction plot – off by default; offered once two or more factors are selected – factorial independent, repeated measures factorial or mixed designs: for every pair of factors and every dependent variable, the cell means along one factor’s levels as one trace per level of the other, with a legend – see interaction. Parallel traces mean no interaction, converging or crossing ones that one factor’s effect depends on the other’s level; the omnibus table says whether the pattern is more than chance, and a three-way interaction stays table-only. A mixed design draws its between × within pairs and the pairs within each side; a design with more factors than the two drawn averages each subject’s repeats over the others. With covariates in the model the traces are the covariate-adjusted cell means of the ANCOVA fit, the title reading Interaction plot (adjusted means), where the fit’s crossing is exactly the pair drawn; a trace missing an interior level breaks there rather than join across the gap. Hovering a mean lists its cell, mean, SD, SE, the bar’s bounds and n.
• Show error bars – on by default: whiskers on every mean spanning what Error bars show selects, with a caption naming which. When one of the pair is a within-subjects factor the bars are the within-subject interval whatever the selector says, save the SD, and a subject missing a cell is dropped from them and counted beneath; a cell of one observation is dropped while bars are on and drawn when they are off.

Forest plot #

• Forest plot – off by default; offered for the numeric families and the One-sample proportion test: with Include effect sizes on, one row per dependent variable – per pair on a pairwise expansion – with the effect size as a diamond, its confidence interval as a line and the numbers annotated beside it, the diamonds sized by precision so the narrowest interval carries the largest. A dashed line marks the measure’s no-effect value – 0 for a difference or correlation measure, 0.5 for the common language effect size, 1 for the variance ratio – and a ratio measure takes a log axis, so a halving and a doubling sit the same distance from 1, falling back to a linear one when a bound is not positive; a measure that reports no interval leaves the plot empty with a note saying so. Two more forests follow where the run has them. With post-hoc tests on under a method that reports a difference with an interval – the exact multivariate t, Tukey HSD, the pooled and the paired pairwise t-tests, Games-Howell, Dunnett’s test, Dunnett’s T3 and lincon – one Forest plot (post-hoc) per dependent variable draws the differences on the variable’s own scale with the line at zero, needing no effect size; the rank-based and categorical methods (Dunn’s, Conover’s, Nemenyi and the Wilcoxon, McNemar and Jensen-Shannon pairs) report no difference and draw none. For Factorial ANOVA, Mixed ANOVA and Repeated measures factorial ANOVA the post-hoc contrasts are forested per effect – a panel per main effect with its pairwise differences and per interaction with its simple-effect contrasts; ART-ANOVA and its repeated measures form have no raw scale to forest and draw none.
• Combine effects into one forest chart – shown for those three families while the forest plot and post-hoc tests are both on: off by default, one panel per effect; on, every effect stacked into one chart under bold effect headings on a shared axis.

Discriminant function plot #

• Discriminant function plot – on by default once Include descriptive discriminant analysis is checked, for MANOVA and MANCOVA: one plot per card, since a function combines every dependent variable, which the per-variable plots cannot show – one point per case at its scores on the canonical functions and each group centroid as a large ringed marker. With two or more functions it is a scatter of function 1 against function 2, each axis labelled with the function’s share of the between-group variance (Function 1 (78.4% of variance)) and a legend naming the groups; with one function – two groups – the y-axis carries the groups instead, each group’s cases spread along its own band. Dashed lines cross at zero, the whole sample’s centroid, and each function is scaled to within-group variance 1 on equal pixel scales, so a gap of 1 on either axis is one within-group standard deviation. Hovering a centroid gives the group, its scores and its n; the case points carry no tooltip. Above 5,000 cases the cloud is thinned evenly within each group and a note says how many are drawn. Read the centroids’ separation along each axis, not the clouds’ density – overlapping clouds around separated centroids are still a real multivariate difference – and which variables build each axis in the standardized canonical coefficients table.

Correspondence map #

• Correspondence map – on by default once Include correspondence analysis is checked: a card of its own, Correspondence map per dependent variable or Multiple correspondence map for the joint analysis, placing the row and column categories in one space – one colour per series, a legend and a tooltip per point with its mass, inertia and quality – see correspondence analysis. Both axes are labelled with the dimension’s share of inertia, dashed lines cross at the origin – the average profile, where independence would put every category – and the two axes share one pixel scale. Each series is drawn in its own principal coordinates, the symmetric map, and its caveat is printed beneath: a distance within a series reads directly, one between two series does not – read across series by direction from the origin. A one-dimensional solution bands the categories along the y-axis by series; labels are placed by inertia and the least-inertia ones dropped where they would overlap, a note counting them, while the coordinate tables name every category.

ROC curve #

• ROC curve – on by default once Include classification (ROC) analysis is checked for an eligible test: per dependent variable, sensitivity against 1 − specificity – one curve per group pair on a pairwise expansion – with the chance diagonal and a legend giving each curve’s AUC – see ROC curve and AUC; the predicted group is read off the classification table. With Classification metrics at optimal threshold on, dots mark the reported thresholds; hover one for its value, sensitivity, specificity, PPV, NPV, accuracy and, under the cost-weighted rule, which error it treats as costlier. With Compare AUCs on, an extra plot overlays every variable’s curve for the same group pair, so the AUCs DeLong’s test compares share one axis.

Category proportion plot #

• Category proportion plot – on by default for the categorical tests, in place of the distribution plots, since a contingency test has no distribution to draw: a Category proportions card with one chart per dependent variable showing how its categories divide within each level of the grouping factor – or each condition, for the paired tests – so a shift in composition is seen rather than inferred from the χ². The corner n gives each group’s total; hovering a bar gives its share, count, group total and z. Under the Cochran-Mantel-Haenszel test a chart per stratum follows the pooled one, titled with the stratum’s value, so whether the association points the same way in every stratum is on view. Above 25 categories the grouped bars would be sub-pixel, so each group becomes one 100%-stacked bar; the tail is never folded into an Other slice. In batch and pseudo-factorial runs the card is labelled with its grouping variable.

Where the association lives: the ▲ / ▼ marks. Bars that stand out carry a triangle, and what it measures follows the test’s shape; a note under the chart names the rule, the critical value and the family it was adjusted over, and hovering a bar shows its z whether or not it cleared the threshold.

• Chi-square test of independence – the bar’s own standardized residual: ▲ more observations than independence expects, ▼ fewer, beyond the critical |z| Bonferroni-adjusted over the table’s cells – the same rule and value that print a residual in bold in the contingency table, so chart and table never disagree.
• McNemar’s test and Stuart-Maxwell test – both bars of a category are margins of one paired table, so the quantity is the category’s: the standardized change in its share from the first condition to the second, drawn as a connector across the pair with the glyph at its midpoint – ▲ a rise, ▼ a fall – against a critical |z| adjusted over the categories.
• Cochran’s Q test – each bar’s departure from its category’s average across the conditions, scaled so that one category’s squared departures sum to Q: ▲ above average, ▼ below, against a critical |z| adjusted over the conditions.

Fisher’s exact test, Cochran-Armitage and the Cochran-Mantel-Haenszel panels compute no such quantity and draw unmarked bars.

Observed vs expected plot #

Drawn for the Chi-square goodness-of-fit test alone, with no checkbox, in its own Observed vs expected card – one chart per variable: each category’s observed count as a bar and its expected count under your expected distribution as a marker overhanging the bar, so the two stay distinct when they nearly coincide; a category added in the editor that never occurred is a marker over no bar, the gap the test weighs. Hover a category for O, E and its contribution to the statistic, ${(O-E)}^2/E$; the bars that stray furthest from their markers, taken together, are what makes the test significant – the results table gives the verdict, the chart says where it comes from.

Reporting checklist #

Key things to include when writing up comparison results:

Method:

• The analysis design – independent, dependent, mixed, or one sample – and, for a dependent or mixed design, that the test ran on the complete subjects, with the number excluded
• The statistical test used and why (e.g. “Welch’s t-test was used due to unequal variances”); with several grouping or condition variables, the analysis strategy – one run per factor, one combined factor, or a native factorial test
• Whether assumptions were checked, which were met or violated, and what a failure changed – the test the recommendations demoted, the correction the verdict was read from
• How missing data were handled
• The p-value adjustment method, if any
• For post-hoc tests: which method and, for the pairwise tests that defer to the global setting, which correction
• For the chi-square, McNemar, Mann-Whitney, Wilcoxon signed-rank (paired and one-sample), Stuart-Maxwell and Cochran-Mantel-Haenszel tests: whether the continuity correction was applied – and, for McNemar’s test (which Stuart-Maxwell’s runs on a binary outcome) with fewer than 25 discordant pairs, that the exact binomial test was used
• For a test whose p came from an exact, permutation or simulated branch rather than the asymptotic one – Fisher’s exact test beyond 30 cells, the goodness-of-fit test on thin expected counts, the Kolmogorov-Smirnov, Anderson-Darling, Jonckheere-Terpstra, Brunner-Munzel, Page’s, Jensen-Shannon and energy tests, Cochran-Mantel-Haenszel on a sparse 2×2×K table – which branch it was and, for a simulated or permutation p, the replicate count; the note under the result records both
• For the goodness-of-fit test: the expected distribution and where it came from (uniform, a theoretical model, published proportions), the number of categories, and whether any expected count fell below 5
• For the sign tests (the paired sign test and the one-sample sign test): that only the direction of each change was used, so no assumption was made about the shape of the differences, why you chose it over the signed-rank test (usually a failed symmetry check), and how many zero differences were dropped as ties
• For the one-sample proportion test: the hypothesised proportion π₀, which category was counted as the success (and that every other category counted as a failure), and that the p-value is the exact binomial one rather than a χ² approximation
• For the parametric within-subjects families (repeated measures ANOVA, repeated measures factorial ANOVA, mixed ANOVA): which within-subjects approach was used – univariate (ε-corrected) or multivariate (Pillai) – and, on the univariate route, which epsilon correction the verdict was read from; on the multivariate route, any term that had to fall back to a univariate F
• For a covariate design whose homogeneity-of-regression-slopes check failed: that the heterogeneous-slopes follow-up is reported in place of a single adjusted difference, and which covariate the regions are read over
• For ART-ANOVA: that the Aligned Rank Transform was used as the non-parametric factorial or mixed-design analysis, and that the denominator df came from the Kenward-Roger approximation (hence the non-integer error df)
• For the variance (dispersion) tests: which test and why (e.g. “Fligner-Killeen was used because the groups were non-normal”), and that the hypothesis is about spread, not location
• For the robust trimmed-means tests (Yuen, t1way): the trim level used (e.g. 20% per tail), and that trimmed means were compared because the data were non-normal and/or heteroscedastic
• For the trend tests (Jonckheere-Terpstra, Page’s, Cochran-Armitage): that an ordered (ordinal) factor was used and how it was ordered, the trend direction tested, the effect size reported (Kendall’s τ-b, or the mean within-block ρ for Page’s), and – for Cochran-Armitage – which outcome category was the modelled event and that the factor’s values served as trend scores
• For the Cochran-Mantel-Haenszel test: the stratifying variable and number of strata, whether the continuity correction was applied (2×2×K), and – for 2×2×K – the Breslow-Day homogeneity result that justifies (or cautions against) reporting a single common odds ratio
• Directionality (one-tailed or two-tailed), and that a one-tailed direction was fixed before the data were seen
• For equivalence testing: the type (TOST, non-inferiority, superiority, or MET), the equivalence bound Δ, and whether Δ was specified in raw or standardized units – for a standardized bound, the spread that converted it and the raw bound it became, both in the bounds note under the table
• For ROC analysis: the threshold rule (Youden / closest / cost-weighted, with the cost asymmetry ratio and any population-prevalence override if applicable), the AUC CI method (DeLong or bootstrap), and that DeLong’s test was used for any AUC comparisons
• For correspondence analysis: the scope (two-way per dependent variable, or joint MCA), that the map uses symmetric scaling – and therefore that between-series distances are not directly interpretable – and, on an MCA, that the inertia percentages are Greenacre-adjusted and so not comparable with two-way ones
• Where a run resampled: the replicate count and the Bootstrap seed the bootstrap intervals were drawn under – the Reproducibility section says which steps it governs

Results:

• Descriptive statistics per group or condition (means, SDs, sample sizes at minimum) – the summary statistics the card prints
• Test statistic with degrees of freedom (e.g. t(58) = 2.34, F(2, 87) = 5.12) – or, for a test that reports none, what it reports instead: Fisher’s exact test has no statistic, the sign tests report a count
• Exact p-value (or p < .001 for very small values), raw and adjusted where both are shown
• Effect size with confidence interval (e.g. Cohen’s d = 0.65, 95% CI [0.12, 1.18]), the measure named – see reporting an effect size
• For the F-test of variances: the variance ratio with its CI (e.g. F(29, 29) = 1.84, variance ratio 1.84, 95% CI [0.88, 3.86])
• For the robust trimmed-means tests: the test statistic with df, the explanatory measure ξ (with its CI for t1way), and – for Yuen – the trimmed-mean difference with its CI
• For the one-sample tests: the reference value μ₀ and the location estimate with its confidence interval – the mean (t-test), the pseudo-median (Wilcoxon), or the median (sign test)
• For the sign tests: the number of positive differences (or of observations above μ₀) out of the effective n, the median with its distribution-free confidence interval and the coverage that interval actually attained, and the effect size r₊ = 2p̂ − 1
• For the one-sample proportion test: the observed proportion p̂ with its Clopper-Pearson interval, the exact p-value against π₀, and Cohen’s h with its interval
• For a repeated-measures or mixed design: the ε and the corrected df the verdict was read at when sphericity failed, and, beside the condition means, whether their interval is the ordinary or the within-subject one
• For the heterogeneous-slopes follow-up: per contrast, the slope difference with its SE, t and p, the Johnson-Neyman region boundaries (noting that only boundaries inside the observed covariate range are reported), and the share of cases falling inside the significant region
• For the Cochran-Mantel-Haenszel test: the MH χ² with df and p-value, and – for 2×2×K – the common odds ratio with its CI and the Breslow-Day χ², df, and p (e.g. CMH χ²(1) = 11.43, p = .001; common OR = 3.00, 95% CI [1.62, 5.55]; Breslow-Day χ²(2) = 0.00, p = 1.00)
• For equivalence testing: the equivalence p (p (equiv) or p (MET)) and the two one-sided p-values, plus the raw Δ bound used and the (1 − 2α) interval of the difference
• For multi-group tests: omnibus result first, then post-hoc comparisons, with the adjustment named
• For factorial/mixed designs: main effects, interactions, and simple effects where relevant, read from the adjusted means
• For ROC analysis: AUC with CI per variable (and per pair, if applicable), plus sensitivity and specificity at the reported threshold; for AUC comparisons, Δ AUC, Z, and p-value(s)
• For correspondence analysis: the total inertia, the number of dimensions retained and the percentage of inertia each carries, and – for any category you interpret from the map – its contribution and quality (the squared cosine), so the reader can see it is well enough represented to bear the reading

Reproducibility #

Every analysis prints the underlying R code to the R console – you can inspect, copy, or re-run the exact commands. Comparison analysis uses base R for the t-tests, the Wilcoxon, sign and exact binomial tests, Kruskal-Wallis and Friedman, chi-square (chisq.test, the goodness-of-fit test included), McNemar’s test, Welch’s one-way ANOVA (oneway.test), the Kolmogorov-Smirnov test (ks.test), the variance tests (var.test, bartlett.test, fligner.test), the Cochran-Armitage trend test (prop.trend.test) and the Cochran-Mantel-Haenszel test (mantelhaen.test); car for the ANOVA family and for the assumption check’s Levene and sphericity tests, with biotools for Box’s M; emmeans for the estimated marginal means and the post-hoc contrasts of the factorial, covariate and within-subjects fits; mvtnorm for Dunnett’s test and, through emmeans, the exact multivariate-t post-hoc; dunn.test for Dunn’s test; DescTools for the Breslow-Day check and Page’s exact null distribution; brunnermunzel, kSamples, clinfun and onewaytests for the Brunner-Munzel, Anderson-Darling, Jonckheere-Terpstra and Brown-Forsythe tests; WRS2 for Yuen’s test, t1way and the lincon post-hoc; energy for the energy test; FNN for the nearest-neighbour distances behind the information-theoretic tests; ARTool for ART-ANOVA, with lme4 and pbkrtest behind it for the mixed-model fit and the Kenward-Roger degrees of freedom; effectsize for the ANOVA family’s η², ω² and ε²; pROC for the ROC block and DeLong’s test; and ca for the joint multiple correspondence analysis. Page’s L and its normal approximation, the Stuart-Maxwell and Cochran’s Q tests, the Ross mutual-information estimator and its permutation test, the two-way correspondence decomposition, the Games-Howell, Conover, Nemenyi, Dunnett’s T3 and Dunn’s-vs-control post-hocs, the equivalence tests and most of the effect sizes with their intervals are computed in the module’s own R code, with no further package. Which seed governs which step: the bootstrap intervals – the effect sizes’ (Wilcoxon r, the matched-pairs rank-biserial, Kruskal-Wallis ε² and η²_H, Friedman’s W, Kendall’s τ-b, the mean within-block ρ, Cohen’s g, Cochran’s average φ², ξ’s interval) and the ROC block’s – run under Bootstrap seed, an empty one giving fresh draws each run; every permutation or simulated p – Fisher’s and the goodness-of-fit test’s simulated p, the Anderson-Darling, Jonckheere-Terpstra, Jensen-Shannon and energy tests’, the symmetry check’s – and the Monte-Carlo integration behind Dunnett’s test and the exact multivariate t draw under fixed seeds of their own, so they repeat whatever the settings say; and ξ’s balanced subsample takes the Reproducibility seed. Citations for the R packages and the statistical methods used in your analysis appear automatically at the top of the output section – so a run cites the tests, post-hocs, effect sizes and corrections it actually named, not just the packages that computed them. The decisions behind the module – which fit stands behind a card, where a threshold comes from, what a run refuses or collapses, why a p-value’s seed is pinned – are argued in its method notes.

Common pitfalls #

Checking assumptions after seeing the results. Click Check assumptions before Run comparison analysis, not after. If you run a t-test, get a non-significant result, then switch to Mann-Whitney hoping for significance, you are inflating the Type I error rate: the check should determine the test, not the other way around – see when an assumption fails.

Ignoring effect sizes. A significant p-value with a tiny effect size (d = 0.05) means the groups are “statistically different” and the difference is practically meaningless; a non-significant result with a moderate effect size (d = 0.50) may only mean the sample was too small to show it. Keep Include effect sizes on and report the measure with its interval beside every p – see why a p-value is not enough.

Multiple grouping variables as separate batch analyses when you need interactions. Whether the effect of treatment differs by gender is a question about the interaction, and Analyze each variable separately – one run per factor – never tests one; combining the factors into a single factor drops the term too. Pick a native factorial test – factorial ANOVA, its repeated-measures and aligned-rank forms, or a mixed ANOVA when one factor is within-subjects – see separate, combined, or native?.

Using dependent-samples tests on independent data (or vice versa). Comparing pre-test and post-test scores with an independent t-test instead of a paired one treats the two sets of scores as coming from different people and wastes the power that comes from knowing each person’s change; pairing rows that are not the same people invents a dependence the data do not have. The design follows from how the cases are related, and a dependent design needs the Subject ID that says which rows are one case.

Fixing the expected distribution to match the data. The goodness-of-fit test only means something when the expected distribution comes from outside the sample – a theoretical model, a fairness assumption, or proportions published elsewhere. Reading the observed split off the data and then entering it (or something close) as the expectation guarantees a non-significant result that proves nothing: decide the weights from your hypothesis before you look at the counts. And take the expected-count note seriously – the app switches to a simulated p when the counts break Cochran’s rule, but a table that thin is thin evidence whichever p is printed; collapse rare categories or collect more data.

Reporting one Cochran-Mantel-Haenszel odds ratio when the strata disagree. The common odds ratio summarises every stratum with a single number, which is only honest when the per-stratum odds ratios point the same way and are similar in size. When the Breslow-Day test rejects – or the per-stratum tables visibly disagree, the association reversing or vanishing in some – the pooled ratio hides that heterogeneity: report the per-stratum results and consider whether the stratifying variable is an effect modifier rather than a confounder. And never stratify by a variable that lies on the causal path between the group and the outcome – holding a mediator fixed distorts the very association you are measuring.

Running a trend test on a factor that isn’t really ordered. The trend tests assume the factor’s levels have a genuine order carried by their numeric values – dose, grade, time. Marking an arbitrary code (group IDs 1/2/3) as ordinal just to unlock the test tests a trend that has no meaning, and the module cannot catch it – it cannot tell a real order from an accidental one; Cochran-Armitage also reads the values as trend scores, so their spacing matters as well. Mark a factor ordinal only when its ascending values truly define the trend you care about.

Claiming equivalence from a non-significant result. A test that fails to reach significance does not show the groups are equal – it shows you could not prove they differ – and negligible is a claim only an equivalence test makes: set the Test direction to one of its equivalence forms. The distinction matters most in clinical research, where “no difference detected” and “proven equivalent” have very different regulatory implications.

Setting the equivalence bound after seeing the data. The bound Δ is chosen before the analysis, from what counts as a practically meaningful difference in your field; a bound picked after the results – just wide enough to clear – invalidates the test, exactly as choosing a one-tailed direction after seeing the means does – see equivalence margin. Pre-register the bound when you can.

Reading too much into a high AUC on a small sample. AUCs have wide confidence intervals at small sample sizes, and the default DeLong interval rests on asymptotic theory. With fewer than about 30 observations per group, prefer the bootstrap AUC confidence interval and treat the point estimate as provisional: an AUC of 0.85 with a 95% CI of [0.55, 1.0] is not a strong signal – it is a wide range that happens to include “excellent”.

Treating PPV and NPV as universal. PPV and NPV depend on the prevalence of the predicted group: a sample balanced 50/50 against a real-world prevalence of 5% puts the reported PPV far above what deployment would see. Sensitivity and specificity are prevalence-independent and generalize; the predictive values describe your sample unless you enter the population prevalence, which recomputes them at the base rate you name.

Choosing the cost direction after seeing the cost-weighted thresholds. The cost-weighted rule reports two thresholds, one for each direction of the cost asymmetry. Picking the row that “feels right” after seeing the numbers – rather than committing beforehand to which error is the worse – is the same post-hoc tuning as flipping a one-tailed direction after the fact.