Comparison analysis – method notes #

The decisions behind the comparison analysis module – which fit stands behind a card, where a threshold or a cap comes from, and what a run refuses, drops or collapses before it starts. Sections follow the manual’s.

Setting up #

A variable fills one design role. A variable placed in two roles – the same column as an ANCOVA’s factor and its covariate, say – builds an aliased model: dv ~ … + X + grouping with the grouping a copy of X, which lm() resolves by silently dropping a coefficient, with a warning that reaches the R console and nothing else. The run refuses the overlap up front and names the variable, since the role derivation on its own only keeps a multi-role variable out of the dependent-variable pool.

Covariates are offered from the numeric variables alone. A covariate enters lm() and car::Anova() as a continuous adjustment; a categorical one would be dummy-coded into an extra crossed factor – a different design from the ANCOVA or MANCOVA adjustment the role promises, and one the card would not label as such – so the list is filtered rather than the model reinterpreted.

A repeated measures ANOVA with covariates is fitted as a repeated measures MANOVA and reported as the univariate test. The one-way repeated measures runner fits aov() with an Error() stratum, which admits no covariate; the repeated measures MANOVA runner fits lm() under car::Anova()'s intra-subject design, which does. A covariate therefore reroutes the run to the second fit, and the card reports what was selected – the univariate within F with its Greenhouse-Geisser and Huynh-Feldt corrections, the effect size and its interval, the covariate effects – while the multivariate omnibus is discarded rather than restated: it is the sphericity-free test over the (conditions − 1) contrast dimensions, a different statistic at any number of dependent variables. The rerouted path has no multivariate route of its own, so the Within-subjects approach control hides for it rather than let a selection be ignored, and the post-hoc condition contrasts come from emmeans on the covariate-adjusted fit.

A dependent variable of the wrong type is dropped with a warning, not refused. A test that reads numeric dependent variables only (or categorical only) filters the pool to those it can run on, and the run is refused only when nothing survives: a selection that mixes types is the ordinary case when every unassigned variable is a dependent variable. A partial filter would otherwise be invisible – the survivors run and the rest vanish – so the warning names the excluded variables and the test’s rule. The goodness-of-fit test admits ordinal variables to its categorical pool, since it reads a dependent variable as a frequency table; the location tests route an ordinal variable to the numeric pool.

The groups preview’s thresholds. A group of fewer than 3 observations is flagged, since two values give one degree of freedom for its spread and one value none, and a group under 10 is muted as a caution; the listing stops at 30 groups and 10 warnings so a wide crossing does not scroll the panel away. A numeric grouping variable with more than 50 distinct values is flagged as probably continuous, since a factor with that many levels is almost never one; the same rule warns at run time about a stratifying variable, which has no preview. A crossing is capped at 10,000 cells – a hard limit so the cell tally and the R factor cannot freeze the browser – and warned about at 1,000 as a cost; more than 100 pairwise comparisons draw a warning to adjust p-values, since a family that large expects five false discoveries at α = .05 on null data. In a mixed design a between cell is sized by its distinct subjects rather than its rows, because each subject contributes one row per within-condition and the row tally would report every cell at k times the n the assumption checks use. Empty cells are counted over the crossing and dropped, and the pairwise count runs over the occupied cells alone.

Complete subjects have every condition; repeated rows in a cell are averaged. A within-subjects test runs on the subjects with at least one row in every level of the condition variable – a subject missing a level has no change to measure there and is excluded, and the preview and the card count them. A subject with more than one row in a level is kept: its rows are averaged into one value per (subject, condition) cell before the fit. The alternative – keeping the first row, which is what reshape(), the rank matrix and aov()'s error strata would each do on their own – would print a cell mean over one set of rows beside the within-subject interval the module builds over all of them; averaging marginalises whatever within factor the repeats measure and leaves the test, the descriptives and the interval on one set of rows. The number of averaged subjects is disclosed in a warning.

An incomplete within-subjects crossing is collapsed into one factor rather than refused. Two or more condition variables whose levels are not fully crossed – some combination has no rows – leave the intra-subject model matrix rank-deficient: car::Anova() refuses the fit outright and ARTool drops the aliased terms without saying so. The module instead treats the observed cells as the levels of a single within-subjects factor – the same within-subject space, only its decomposition into main effects and interaction dropped – which is the fit the design supports; the preview names the missing combinations before the run and the card discloses the collapse after it. The collapse takes the tests that cross within-subjects factors: repeated measures factorial ANOVA, its ART form and the mixed family.

The strategy is offered to independent and dependent designs on a single-factor test alone. A native factorial test consumes every selected factor in one model, so there is nothing to iterate or to combine and the radios hide; a mixed test crosses its between and within factors through the same intra-subject design and always runs as one fit. The trend tests get Analyze each variable separately only: Combine variables into a single factor would hand them a nominal crossed column, and a trend statistic reads its factor’s numeric order, so the crossing would rank an arbitrary cell order and return a confident-looking wrong answer. Under the separate strategy the runs’ p-values are pooled into one family for the global adjustment, since they are tests on the same data.

Choosing a test #

A precondition refuses the run; only the analysis type hides a test. The select filters on the design alone – independent, dependent, mixed or one sample – and every other requirement is checked when the run starts, with a message naming what to change. A test that vanished from the list when its precondition lapsed would be undiscoverable: a user who never selected two dependent variables would never learn the MANOVA family exists, and one whose factor is nominal would not learn that a trend test wants it ordinal. The pre-flight runs before any R call, so a refusal costs nothing, and it covers the programmatic and restored configurations the panel’s own hiding never sees – a one-tailed direction on an omnibus, an equivalence direction on a test without TOST plumbing, a two-sided trend on Page’s test – so that a stale setting can never reach a runner that would silently misreport it.

A two-sample test on more than two levels expands to every pair, without an omnibus. Each pair is tested directly and the pairs’ p-values join the run-wide family for the global adjustment, which is where the multiplicity is controlled; an omnibus gate would add a second decision the adjustment already makes. The error term follows the variance assumption the chosen test makes. The independent samples t-test recomputes each pair’s t from the standard deviation pooled over all the groups, with N − k degrees of freedom – pairwise.t.test(pool.sd = TRUE), a Fisher’s LSD contrast – which borrows strength from the whole sample and is valid precisely because the test already assumes equal variances; Welch’s t-test keeps a separate-variance t per pair, since pooling would re-import the premise it drops; every other two-sample test – the rank tests, Yuen’s, Kolmogorov-Smirnov, the F-test, the paired and paired-categorical tests – runs each pair on its own data, as pairwise.wilcox.test and pairwise.t.test(paired = TRUE) do.

Welch’s t-test reports Welch’s d and g, not the pooled-SD Cohen’s d. Cohen’s d and Hedges’ g standardize by the pooled standard deviation, which contradicts the unequal-variance premise that selects Welch’s test in the first place; Welch’s d standardizes by a variance-appropriate standardizer and Welch’s g is its bias-corrected sibling – the pairing Student’s t offers as Cohen’s d and Hedges’ g, and the standard recommendation for Welch (Delacre et al. 2021) – while Glass’ Δ uses one group’s SD by construction. η² and ω² are omitted too: computed from a Welch t with fractional Satterthwaite df, they yield a variance explained that maps to no real partition. The point-biserial r is reported as the homoscedastic correlation, standardized by a reconstructed pooled (N − 2) SD whichever test ran.

A trend test runs on an ordinal factor with three or more levels, or not at all. Jonckheere-Terpstra, Page’s and Cochran-Armitage read the trend from the factor’s ascending numeric values; there is no in-app reordering step, so a nominal factor would be ranked in an arbitrary level order and return a confident-looking wrong answer, and the run refuses it with the variable to recode. At two levels the trend reduces to a plain two-group test, so three is the floor. The combine-into-one-factor strategy is refused for them because a crossed nominal column has no order – the Setting up note above says why the radio hides – and Page’s L tests a specified ordering and has no two-sided form, so a two-sided direction reaching it from a stale configuration is refused rather than silently computed as the increasing trend.

Jonckheere-Terpstra takes a permutation p on ties or at n ≤ 100. clinfun::jonckheere.test uses its exact jtpdf path for n ≤ 100 without ties, and that Fortran routine is broken under WebAssembly – the distribution does not sum to one, so p can fall outside [0, 1]; with ties it drops to a power-losing normal approximation. The runner forces the package’s seeded permutation p in both cases – n ≤ 100 is the exact path’s domain, and ties occur above it too – under the permutation replications setting, disclosing the count; n > 100 without ties keeps the package’s normal approximation, which is clean there. Kendall’s τ-b is the companion measure because it restates the same monotone association on a −1…1 scale and is computable on the permutation branch too.

Cochran-Armitage scores the factor by its own values and models the higher outcome category. prop.trend.test’s default scores are equally spaced, which would test a dose of 1 / 5 / 10 as three equal steps; the runner passes the ordinal factor’s actual numeric values, aligned to the table’s columns by level index, and returns the fitted vector so the card can disclose it. The 1-df trend χ² is re-expressed as a signed z – its square – so the trend direction control can ask for an increasing, a decreasing or a two-sided trend, and the event is the higher of the two outcome codes, named in a note, since the sign of z is meaningless without it. Cochran’s expected-cell rule is checked on the 2 × k table as for the other χ²-derived tests, and a binary dependent variable is pre-flighted rather than left to the runner’s error.

Brunner-Munzel takes the exact permutation branch while the label splits can be enumerated. Neubert and Brunner’s (2007) permutation test enumerates every division of the pooled sample into the two group sizes, so it is exact but bounded by $\left(\genfrac{}{}{0px}{}{n_1+n_2}{n_1}\right)$; the package’s own feasibility cap is 4 × 10⁷ splits, but the Fortran enumerator allocates per split and traps the WebAssembly heap past about 1.3 × 10⁵ – C(20, 8) = 125,970 runs, C(20, 9) = 167,960 kills the engine outright and the page has to be reloaded. The runner therefore takes the permutation branch at or below 100,000 splits and the Satterthwaite approximation above it, which is the regime the approximation is accurate in anyway, and small samples – where the approximation is least dependable – get the exact p. The permutation branch returns neither a statistic nor degrees of freedom by design, so those columns stay blank and a note names the branch.

Kolmogorov-Smirnov: an exact p below n₁·n₂ = 10,000, two-sided only, with D⁺ and D⁻ in place of a one-tailed p. ks.test computes the exact distribution of D, ties included, while the product of the group sizes is under 10,000 and switches to the asymptotic approximation above it – R’s own threshold – which the card discloses rather than letting the switch pass silently. The test direction control is withheld because ks.test(alternative = "greater") means that group 1’s distribution function lies above group 2’s – group 1 stochastically smaller – the opposite of what the shared Group 1 > Group 2 label asserts for every other test on that control. The card reports D⁺ and D⁻ with the value each is attained at instead, taken off the two ECDFs, which answers the directional question with a location a one-sided p lacks; the two-sided p is conservative for a directional reading, since in the tail $P(D>d)\approx 2P(D^{+}>d)$, and the statistic note says so.

Anderson-Darling simulates its p on ties or a group under 5, and refuses a pooled sample it cannot rank. kSamples::ad.test makes no adjustment for ties in its asymptotic p and sets its own warning flag when a group holds fewer than 5 observations; either invalidates the asymptotic p, so the runner reruns the test with the package’s simulated group-label permutation under a pinned seed and the permutation replications setting, and discloses which condition fired and the count. The simulated p is recovered as $(1+r)/(N_{\text{sim}}+1)$ from the package’s raw $r/N_{\text{sim}}$, so a well-separated sample reports the floor its replicate count actually bought rather than 0. The statistic reported is the standardized T.AD of the package’s second row – the tie- and discretization-robust version – not the raw AD statistic, which would misread against published critical values. A pooled sample of two observations, or one where every group is a singleton, makes the package return NA with p = 1, which would read as a confident no difference; the run reports an error instead.

The heteroscedastic and robust omnibuses report ξ and no variance-explained measure. η², ω² and ε² partition a total sum of squares that exists only under equal variances – the premise Welch’s ANOVA, the Brown-Forsythe F* and t1way drop – so a variance explained computed from them maps to no real partition, exactly as the pooled-SD d does for Welch’s t-test. The explanatory measure ξ takes their place: a ratio of between-group spread to pooled winsorized spread that assumes no partition, evaluated at γ = 0 for Welch’s and Brown-Forsythe’s, the trim of the contrast those omnibuses make, and at the run’s trim level for t1way and Yuen’s, whose estimand is the trimmed-mean difference. Yuen’s adds δ_t, the signed standardized trimmed-mean difference, because ξ is unsigned and carries magnitude without direction. The same family’s post-hocs are Games-Howell and Dunnett’s T3 – separate-variance methods that need no pooled term – and lincon at the omnibus’s trim. Johansen’s MANOVA sits on the same reasoning one level up: η² and its siblings partition a sum of squares that exists only under a pooled covariance.

Johansen’s MANOVA is refused where a group’s own covariance matrix cannot be inverted. Each group is weighted by its own covariance matrix (Johansen 1980), which is what makes it the multivariate Welch’s test: on unbalanced designs with a 3× SD imbalance it holds nominal size (.052 / .050) where Pillai’s trace gives .406 when the small group is over-dispersed and .002 when the large one is, which is why the Box’s M demotion steers to it. It still assumes multivariate normality and is demoted on that failure exactly as the pooled family is. Estimating a matrix per group has three consequences the run enforces up front rather than inside the fit, where solve() on a singular matrix would fail with a raw R error: every group needs more complete observations than there are dependent variables, the dependent variables must stay linearly independent within every group – the message names the groups at fault and points to MANOVA, which pools one matrix – and there is no pooled error term for a contrast of the multivariate omnibus, so the omnibus carries no effect size and the pairwise comparisons run on the Welch’s ANOVA follow-ups it always emits; a pooled follow-up would re-import the assumption the omnibus was chosen to avoid.

The energy test standardizes, trades replicates down against a fixed work budget, and refuses past it. Székely and Rizzo’s k-sample test (energy::eqdist.etest) is distribution-free, so it is the fallback the multivariate-normality demotion steers to: on lognormal margins it holds nominal size (.054 / .040 / .032 at 10 / 20 / 40 per group) where Pillai’s trace collapses to .020. Its null covers spread and shape as well as location – a pure difference in dispersion rejects at close to 1.00 – so a rejection is not evidence that the means differ, which is why the name says “equal distributions”, why a significant Box’s M demotes it too, and why it is offered as a fallback and not a MANOVA substitute. The E-statistic is built from raw Euclidean distances and is not scale-invariant the way Pillai’s and Johansen’s are – measured on the live engine, a 100× rescale of one irrelevant variable drove power from .93 to .03 – so the dependent variables are standardized first; scale() uses the whole sample, which is invariant to the label permutation, so the test stays exact. The permutation runs inside C and never yields, so neither the engine’s interrupt nor the Cancel button can reach it, and the run must be bounded up front: the cost is close to linear in tests × replicates × n² and independent of the number of variables, 0.8–2.9 s at 10⁹ across the tested grid, so the budget is 10⁹ units shared between the omnibus and one follow-up per variable, all at the same cost. Replicates are traded down to fit before the run is refused, since a coarser p still answers the question, and the count used is disclosed; below 199 – a resolution of 1/200 – or the user’s own coarser setting, the run refuses, which at two dependent variables is about 1,300 complete rows, and the message names MANOVA and Johansen’s, whose normality advantage this test holds at small n is gone by roughly 500 rows, and the replications setting. There is no post-hoc block because there is no pooled term to contrast; the follow-ups are per-variable energy tests, which share the omnibus’s null and assume nothing it does not, reported as E-statistics with no F and no df. E itself grows with the sample and the scale of the distances, so the effect size is R²ₑ, the DISCO partition of the same total dispersion into its between and within parts, computed at R = 0 so it costs nothing against the budget.

The within-subjects multivariate fits need residual degrees of freedom above the within-contrast dimension. Repeated measures MANOVA fits lm() under car::Anova()'s intra-subject design, and the within-subject test exists only while the residual degrees of freedom – subjects − 1 − covariates – exceed the dimension of the within contrasts, dependent variables × (conditions − 1); below it the error matrix is singular. Mixed MANOVA’s bound is subjects − between-cells − covariates against the largest within term, dependent variables × ∏(condition levels − 1), and a repeated measures ANOVA rerouted through the MANOVA fit by its covariates meets the same bound at one dependent variable. The check runs before the fit, with both numbers and the three ways out in the message, and runs again after the subjects with missing values are removed, since the complete-subject count is what the fit sees.

An empty between-subjects cell is refused under Type III and fitted under Type II. A fully empty cell of the grouping crossing aliases a coefficient of the interaction, so car::Anova(type = 3) throws “aliased coefficients” for the whole effects table; Type II builds its main effects from model comparison and only the interaction row goes missing, so the run blocks Type III with a message that names the cell and the Type II route, and lets Type II proceed with the warning that the interaction is not estimable. The check reads the crossing, not the level counts, which the per-factor guards already cover, and stops on the cell cap first, since an over-cap crossing returns no cell list to scan. It applies to every factorial and mixed test with two or more grouping variables, the ART forms included – for those, which offer no sums-of-squares choice, the message asks for a factor to be dropped or collapsed.

A covariate, or a mixed design’s grouping variable, takes one value per subject. The covariate-adjusted within-subjects runners fit one wide row per subject, so a covariate that varies within a subject would either be refused inside the fit or, worse, silently pseudo-replicate; the run rejects it up front with the variable named, and only for tests that consume covariates – ART ignores them and discloses it, so rejecting a config over a variable the runner drops would block a valid analysis. A mixed design’s between factor is subject-constant for the same reason: the model fits one between label per subject, and the reshape, the mixed Box’s M and the mixed ANOVA fit would each silently keep whichever label appeared first.

The information-theoretic tests are independent-samples only, on a k-NN estimator, by permutation. Mutual information and the Jensen-Shannon divergence are estimated with Ross’s (2014) k-nearest-neighbour estimator for mixed discrete–continuous data rather than by binning, with a tiny seeded jitter breaking exact ties so that a within-group neighbour radius never collapses to zero and inflates the estimate; the p is a group-label permutation over the permutation replications setting, which is why the count is disclosed. Both are omnibus quantities over all k groups – the generalized Jensen-Shannon divergence (Lin 1991) compares every group’s distribution against their mixture and is bounded by log₂ k, a genuine k-group quantity rather than a pair in disguise – so the pairwise breakdown is an opt-in post-hoc that re-runs the same estimator per pair, each cell bounded by 1 bit. The raw statistics are uncalibrated – bits against a ceiling that moves with the group entropy or the group count – so the effect sizes are their normalizations by that ceiling: Theil’s U for mutual information and the divergence over log₂ k for JSD. Neither is offered for a paired or mixed design, where “did the outcome shift within subject?” has no clean information-theoretic answer.

The sign tests’ interval is read off the order statistics, and its coverage is discrete. binom.test’s own interval brackets the proportion, not a location, so the paired sign test’s interval for the median difference and the one-sample sign test’s for the median come from the order statistics the test is built on (Conover 1999, §3.4): $[d_{(k)},d_{(n-k+1)}]$ covers the median with probability $1-2\text{pbinom}(k-1,n,0.5)$, and the runner takes the largest k whose two-sided tail still fits inside α, so the coverage is at least the level set. A discrete family admits no exact level, so the coverage actually achieved is reported beside the interval; at 95% a qualifying k needs n ≥ 6, and below that no pair of order statistics reaches the level and the interval is reported as the whole line, [−∞, ∞], at coverage 1. Pairs with a zero difference, or observations at μ₀, are dropped as ties before the count, as binom.test on the signs requires, and the effect size r₊ = 2p̂ − 1 is the count’s rescaling to −1…1. Neither sign test offers the equivalence directions: binom.test is not a t- or Wilcoxon-family location test, so the μ-shifted refit the TOST machinery runs has nothing to run on.

Fisher’s exact goes Monte-Carlo past 30 cells, or 1,000 rows on a table with three or more rows and columns. A 2 × 2 table runs the exact hypergeometric test at any size, and the Fisher–Freeman–Halton generalization attempts exact enumeration on larger tables – but FEXACT cannot be interrupted once started, and measured on the live engine a table with three or more rows and columns took 10.6 s at n ≥ 2,000 inside the default workspace, while a 2-row (or 2-column) network stays narrow at any n, 224 adversarial tables out to n = 100,000 peaking at 0.34 s. The exact path is therefore attempted below 30 cells and, once both dimensions exceed two, below 1,000 rows, with the workspace lowered from R’s 2 × 10⁷ to 2 × 10⁵, and falls back to a Monte-Carlo p over 10,000 replicates under a pinned seed whenever those bounds are exceeded or the workspace is exhausted; the card names the simulation, and flags a p at its $1/(B+1)$ floor as an upper bound rather than a point. The bounds are empirical, so widening the fallback is preferred to raising them. One-tailed alternatives exist for the 2 × 2 odds ratio alone – the r × c generalization has no directional form – so the direction control is gated on every axis having exactly two realized categories, and the pre-flight repeats the check for configurations that bypass the panel.

McNemar’s test is exact below 25 discordant pairs, and needs a binary outcome. The asymptotic χ² is unreliable on small discordant counts, so the wrapper mirrors wilcox.test’s exact = NULL behaviour: below 25 discordant pairs it runs the exact binomial test on them (b out of b + c at p = ½), reporting the count in a note, and otherwise mcnemar.test with the continuity correction the user controls; a table with no discordant pairs at all is reported as undefined. The runner needs an exact 2 × 2 and stops otherwise, steering to Stuart-Maxwell, so the Bowker generalization is never reached; the binary requirement is pre-flighted so the round trip is not spent. Stuart-Maxwell’s 2 × 2 reduction runs the same wrapper, so the two agree on a binary outcome and the card names McNemar’s.

Cochran-Mantel-Haenszel pools out singleton strata and switches to the exact conditional test on a sparse 2 × 2 × K table. A stratum holding one observation contributes exactly nothing to the statistic but makes mantelhaen.test abort, so such strata are removed before the fit and counted in a note; a stratum emptied to fewer than two levels on either axis by the missing-value filter cannot be fitted and is reported as degenerate. On a 2 × 2 × K table the Mantel-Fleiss criterion – the expected count of the first cell, summed over strata, at least 5 from both bounds of its support – decides whether the χ² approximation is adequate; when it fails the runner reruns mantelhaen.test(exact = TRUE), the standard remedy, whose conditional-MLE odds ratio and exact interval the effect-size branch then reads, and which has no χ² statistic and no df. The exact test can fail outright on a perfectly separated table, where the MLE cannot be bracketed, in which case the asymptotic result stands with its sparse-table warning; beyond 2 × 2 × K no Mantel-Fleiss criterion exists, so Cochran’s rule is applied to the pooled expected counts instead, at the same bar as the plain chi-square’s. The continuity correction and the common odds ratio are defined for 2 × 2 × K only and are inert or absent on a general I × J × K table.

The goodness-of-fit and proportion tests: a simulated p on thin expected counts, and an interval that nearly always agrees with the exact test. The goodness-of-fit runner checks Cochran’s expected-cell rule on the fitted expectations with the same helper the assumptions tab uses, so the two verdicts cannot drift, and when it is violated recomputes the p by seeded Monte-Carlo simulation over 10,000 replicates – the statistic, df and expected counts stay from the asymptotic fit, whose X² is identical – flagging a p at its floor as an upper bound. A category the sample never contained can be entered and is tested at an observed count of zero, and a zero weight is refused because it makes the χ² undefined. The one-sample proportion test is binom.test on the chosen category’s count, so its p is exact with no χ² approximation, and the interval on the card is binom.test’s own, the Clopper-Pearson construction. The two read the same binomial probabilities but are not one decision: the interval inverts the equal-tailed exact test, α/2 in each tail, while binom.test’s two-sided p sums every outcome no likelier than the observed one, so at the edge the interval can cover π₀ where the p rejects, or the reverse – 34 of the 7,106 (x, n, π₀) combinations over n = 20, 50, 100, 200 and π₀ = 0.05 … 0.95 at α = 0.05, and binom.test(135, 200, 0.6) itself gives p = 0.030 where twice the smaller one-sided p, the reading the interval answers to, is 0.035. Neither is refitted to the other, since each is the standard form under its own name; under a one-tailed alternative binom.test’s own interval is one-sided, so the two-sided interval is refitted for the card. Cohen’s $h=2\arcsin \sqrt{\hat{p}}-2\arcsin \sqrt{{\pi }_0}$ is the effect size, on Cohen’s own scale, its bounds carried through the same arcsine map so they agree with the interval on p̂.

Configuration #

The direction control is declared per test, and the equivalence group where the runner is t.test or wilcox.test. A one-tailed alternative is offered where the R call takes one and the label can name what is ordered – the t-tests, the rank and sign tests, the proportion test, the F-test, whose labels name the variances because a bare “Group 1 > Group 2” beside a dispersion test reads as a location hypothesis, and Fisher’s exact test on a 2 × 2 table, gated as its note above says. Yuen’s test is two-sided because WRS2’s yuen() takes no alternative; Kolmogorov-Smirnov’s control is withheld because ks.test’s “greater” asserts the opposite of the shared label, so its card reports D⁺ and D⁻ instead; an omnibus has no direction. The equivalence directions refit the test with its null shifted by Δ – mu = ±Δ – and so exist for the t.test and wilcox.test runners alone: the sign tests’ binom.test, Brunner-Munzel, Fisher’s, the F-test and the proportion test have no shifted-null form, and the group hides for them; a stale one-tailed or equivalence direction carried in by a restored configuration is refused before any R call rather than silently run two-sided. The labels are re-derived per design – the groups, the two conditions by name once a single two-level condition variable is fitted (in the app’s level order, not the order they were selected in), the variable against μ₀, p̂ against π₀ – so that the option always names the quantity the chosen test orders.

Every equivalence direction is the test’s own call with a shifted null, and a standardized Δ is rescaled by the test’s spread and then held fixed. TOST runs t.test or wilcox.test twice, with mu at −Δ and +Δ (μ₀ ∓ Δ for one sample) under alternative = "greater" and "less", and reports the larger p; the minimal-effect test runs the same two calls with the tails swapped and reports the smaller; non-inferiority and superiority make one call against one bound, the preferred direction choosing which, and the reversed p on the card – the same hypothesis with the groups swapped – is a second fit, not a transform. The difference’s interval is built at 1 − 2α so that “inside the bounds” and “TOST rejects at α” agree. A standardized Δ becomes raw units through the spread the chosen test’s own effect size standardizes by, so that the bound and the statistic share one scale: the pooled SD for the independent samples t-test – the pooled √MSE across all groups under Student’s pairwise expansion – Welch’s $\sqrt{(s_1^2+s_2^2)/2}$ for Welch’s, the SD of the differences for the paired t, the sample SD for the one-sample t, and for the rank tests a MAD scaled to a normal SD (× 1.4826) – pooled by n for Mann-Whitney, of the differences for the signed-rank, of the sample for the one-sample Wilcoxon – with the SD standing in where the MAD is zero, which the card’s note then says. The rescaled Δ enters the test as a constant, TOSTER’s plug-in convention, so the standardizer’s own sampling error never reaches the bounds, which makes them slightly anti-conservative; a raw bound sidesteps it.

The post-hoc checkbox hides at two levels on a one-way design only. A one-way factor with two levels has one contrast, which the omnibus already is; a factorial or mixed design keeps the checkbox at binary factors because its interaction cell-pairs and simple-effect contrasts stay informative, and the mixed designs build their per-effect tables internally rather than through the flat pairwise engine. The energy test declares no post-hoc because it leaves no error term a contrast could be built on (its note above), Johansen’s compares on its per-variable Welch follow-ups for the same reason one level up, and MANOVA and MANCOVA contrast each dependent variable’s univariate follow-up, since a multivariate omnibus has no single difference to pair off.

The exact multivariate t leads Tukey wherever both are offered, and takes the interaction cell-pairs under every self-correcting method. Both contrast the same pairs; the studentized range is exact for balanced cells sharing one error term and conservative once the design is unbalanced, where the multivariate t integrates over the contrasts’ realised correlation – on a balanced design the two agree to rounding, and at group sizes 60/25/60/60 the multivariate t’s p-values are uniformly smaller, by up to 7%. The within-bearing runners drop Tukey for a further reason: each of their contrasts carries its own error term, so the equicorrelated pairwise structure the studentized range is exact for is gone, and the multivariate t is its exact analogue over the realised correlations – repeated measures ANOVA, repeated measures MANOVA and mixed MANOVA offer the multivariate t, Dunnett’s and the deferring pairwise tests alone. Interaction cell-pairs – the difference-of-differences contrasts – are a set Tukey is not defined for, and emmeans silently rewrites tukey to Sidak over them, so under Tukey, Dunnett’s or the multivariate t itself they take adjust = "mvt", and under the pairwise tests the run-wide setting, the one method that defers to it; the ART tests contrast through art.con under the same routing, the select choosing only the correction.

The pairwise t-tests’ intervals widen to the Bonferroni level whenever an adjustment runs. The one-way route recovers each pair’s difference and interval from the pooled MSE beside pairwise.t.test’s p matrix, and when the run-wide adjustment is anything but none the interval’s critical value is taken at α over the number of realized comparisons, so that an interval can never exclude zero beside a non-significant adjusted p; emmeans does the same on the factorial and within routes. No interval is dual to a step-down or false-discovery procedure, so under Holm, Hommel, Hochberg, Benjamini-Hochberg and Benjamini-Yekutieli – all less conservative than the Bonferroni widening – the converse can happen: a row significant while its interval includes zero. Both halves stay valid on their own terms, the p controlling the error rate over the tests and the interval being a valid simultaneous set, and a caveat under the table says so whenever one of those five is active, except for a single-contrast family, where every adjustment collapses to the raw p and the two cannot disagree.

Dunnett’s calibration reaches the within-bearing designs through emmeans, and a displaced control is replaced with a notice. Two routes share the id: the one-way runners hand-roll the pooled-variance t against qmvt and pmvt, and every emmeans-contrasting runner asks for trt.vs.ctrl under adjust = "mvt" – both integrate the exact multivariate t over the k − 1 control contrasts’ realised correlation, and the second is Dunnett’s calibration on a fit whose contrasts each own their error term, which is what lets it reach the repeated measures and mixed designs the pooled t cannot. Dunnett’s T3 keeps each contrast’s separate-variance t and Welch-Satterthwaite df and refers them to the studentized maximum modulus, the heteroscedastic family’s only many-to-one option; Dunn’s vs. control is Dunn’s z over the control contrasts alone, adjusted by the run-wide setting over those k − 1. The control is chosen per factor the run fits and defaults to the first level; when a batch or a filtered fit leaves the chosen level no level of the fitted variable, or one with no usable data, the first level with usable data stands in and the header says which and why, since a substituted baseline read as the chosen one would invert the meaning of every contrast. The three methods always render long, a k × k matrix of k − 1 contrasts being a single populated row.

The continuity correction is on by default and lives only where R applies it. The box is checked so that a card matches R’s and SPSS’s defaults – correct = TRUE in chisq.test, mcnemar.test, mantelhaen.test and wilcox.test – and the Yates correction’s known over-caution once every cell reaches 25 is why the box exists rather than why it is off. The rank tests’ ±0.5 shift fires only on the normal approximation, since exact = NULL picks the exact distribution below 50 per group (50 pairs for the signed-rank forms) when there are no ties, and the exact branch ignores correct silently, as CMH’s exact conditional test does, so the card discloses the branch. The table tests honour the flag on a realized 2 × 2 alone, so the checkbox hides when no selected axis pair can form one and shows as soon as any can – one binary dependent variable beside a binary factor, a two-cell crossing, or a wider factor that post-hoc slicing will cut into pairs – because hiding it forces the flag false and would override R’s default for the tables that are 2 × 2; the tables where the flag was inert are named under the results instead. The pairwise McNemar tests after Cochran’s Q run correct = FALSE whatever the box says, matching Q’s own uncorrected two-condition reduction, with the exact binomial branch below 25 discordant pairs covering the small-sample case a correction would otherwise serve.

The trim level defaults to 20% and is refused at 50%. Twenty percent per tail is Wilcox’s (2017) recommendation and WRS2’s default: enough to bound the influence of the tails while the trimmed mean keeps most of the mean’s efficiency under normality. The entry is a percentage converted to the proportion γ, and WRS2 requires 0 ≤ γ < 0.5 – a 50% trim is the median and leaves no winsorized variance to build the standard error from – so a blank or out-of-range value is refused with a message rather than silently replaced by the default, which would report a number under a trim the user never set. The one γ is threaded to every group, every pairwise Yuen contrast, lincon and the ξ effect size, so that every number on the card describes the same estimand.

The multivariate omnibus reports the Pillai-recovered partial η² whatever the follow-ups’ measure, and a rerouted repeated measures ANOVA swaps its measures before the run. The multivariate effect size is the partial η² recovered from Pillai’s approximate F – V/s, the SPSS value – and the renderer labels it η²p; it tracks only whether effect sizes were asked for, since the η²/ω²/ε² choices read a sums-of-squares decomposition a multivariate statistic does not have, and go to the univariate follow-ups instead, computed there through the effectsize package. The effect-size block hides for a test whose registry lists no measure – Kolmogorov-Smirnov, Anderson-Darling, Bartlett’s, Fligner-Killeen and the energy test, whose R²ₑ is energy::disco’s between-to-total ratio on the omnibus itself, computed regardless. A repeated measures ANOVA with covariates runs as a repeated measures MANOVA (the Setting up note), whose registry names the classical non-partial η² but no partial ω²: generalized η² is mapped to that η² rather than left to the runner’s gate, which would silently land on partial η², and partial ω² – which excludes the subject stratum, whereas the MANOVA runner’s ω² is the classical non-partial value – is rewritten to partial η² before the run, with a warning, so that the column header, the value and the warning name one measure.

The d-family intervals invert the non-central t where the standardizer is pooled and are Wald elsewhere; the r family follows its own variance. Cohen’s d and Hedges’ g standardize by the pooled SD, so t = d/a is non-central t(df, ncp = δ/a) with a = √(1/n₁ + 1/n₂), and the two noncentrality roots at the tail targets map back through d = ncp·a (Steiger & Fouladi 1997; Cumming 2012) – an asymmetric interval bounded by the data, replacing the symmetric Wald-on-t one. d and g estimate the same δ, so the three g variants share the uncorrected d’s interval and the bias correction moves the point estimate alone, where effectsize, JASP and jamovi rescale the bounds by j instead – a legend names the difference rather than let two intervals be read as a defect. Under Student’s pairwise expansion the standardizer is the across-all-groups √MSE on N − k df, the pooled pairwise t-test’s own error term, so d, its interval and the statistic share one standardizer and df. The paired Cohen’s dz and Hedges’ gz invert the same non-central t on t = dz√n with n − 1 df, and so agree with the paired t-test’s own decision about zero; the one-sample d and g do likewise. Glass’ Δ, Welch’s d and g, Cohen’s d_av and Hedges’ g_av keep Wald intervals on the t distribution, because a non-pooled standardizer – one group’s SD, the average-variance denominator, the average of the two condition SDs – admits no clean non-central t; Welch’s d and g sit on the Welch-Satterthwaite df, and the whole interval of Welch’s g and of g_av scales by the bias-correction factor j, where Hedges’ g’s stays put. The Algina-Keselman-Penfield robust d takes a Wald interval on Yuen’s own SE and Welch-type df, carried onto the δ scale. Point-biserial r uses the Fisher-z back-transform on a df-based SE, its interval on the z scale and so no SE beside it. Rank-biserial r and Cliff’s δ are one statistic – δ = 2U/(n₁n₂) − 1 – and share Cliff’s (1993) consistent variance from the per-row and per-column dominance spreads with his asymmetric interval (1996), which stays inside (−1, 1) by construction rather than being truncated at the bounds; the approximate variance stands in only when the consistent one cannot be computed. The common language ES uses the Hanley-McNeil (1982) concordance variance, which accounts for the dependence among the n₁·n₂ pairs the naive binomial form ignores, with a delta-method interval on the logit scale so that it stays in [0, 1] – and no SE, since a symmetric SE on the raw scale would not match the interval beside it.

Variance-explained and χ²-family intervals invert the non-central F and χ²; a repeated-measures η² is a point estimate alone. η², ω² and ε² for the t-tests and the ANOVA family – one-way, factorial, repeated measures, mixed, ANCOVA and the multivariate follow-ups – and partial η² and partial ω² invert the non-central F (Steiger 2004; Smithson 2001): the transform λ/(λ + df₁ + df₂ + 1) bounds the one population proportion of variance all three estimate, so the interval is shared, asymmetric and bounded in [0, 1], matching MBESS::ci.pvaf, a bias-corrected point estimate (ε², ω²) can sit slightly off-centre, and no SE is reported beside an interval of that shape. On the factorial, mixed and repeated-measures tables the η²/ω²/ε² choices come from the effectsize package and are re-based off its denominator – the effect sums of squares plus each error stratum once – onto the corrected total over every observation the fit used, since a repeated-measures ω² adds the subject stratum’s error for every effect. For a repeated measures ANOVA the η² and generalized η² are whole-model proportions whose denominator spans the subject stratum, which the within-effect non-central F does not bound, so they are point estimates alone; the interval lives on partial η² and on partial ω² (Olejnik & Algina 2003), which stay partial. Cramér’s V, McNemar’s φ and Cohen’s w invert the observed χ² against the non-central χ² (Smithson 2003) and map the noncentrality bounds through √(λ/(n·scale)): V and φ are capped at 1, w is unbounded above; V is computed from the uncorrected Pearson χ² even where the test applied Yates’s correction, since the correction is for the p-value, not the magnitude, and its benchmarks are Cohen’s w’s divided by √df, so that a k×k table is not read as a larger effect than a 2×2 with the same association. The variance ratio reports s₁²/s₂² with var.test’s exact F-distribution interval and no magnitude label, since a variance ratio has no canonical benchmark (1 is equal spread). Theil’s U and the normalized Jensen-Shannon divergence carry no interval at all: the k-NN estimator has no analytic one, and a with-replacement resample repeats rows and collapses the radii the estimator is built on.

The proportion-based measures carry the test’s own interval, and the odds ratio three constructions behind one label. The sign-test r is 2p̂ − 1 on the non-tied observations, with a Wilson score interval on p̂ (Newcombe 1998) mapped through the same transform, so that 0 is a median at μ₀. Cohen’s h is 2·asin(√p̂) − 2·asin(√π₀), and its interval is the test’s own Clopper-Pearson bounds put through that monotone transform, so h’s interval excludes 0 exactly when the interval on p̂ excludes π₀. The odds ratio is Fisher’s and Cochran-Mantel-Haenszel’s own estimate and interval where the test object carries one – re-fitted two-sided at the confidence level under a one-tailed Fisher, since fisher.test’s interval follows the alternative and a half-open one would render as ∞ and drop off the forest plot; McNemar’s is b/c, the ratio of the discordant cells, and the chi-square test’s ad/bc, both with Woolf’s (1955) log-scale SE. The Haldane-Anscombe +0.5 is applied to every cell the estimate uses only when one of them is empty – applying it to a full table would introduce avoidable bias – and the corrected estimate carries a ‡; a McNemar table with no discordant pairs at all reports no odds ratio, since the test is undefined there and the correction would manufacture an OR of 1 beside it. No SE is reported, since se(OR) on the ratio scale would not match the log-based asymmetric interval beside it.

A bootstrap interval is a percentile one over the test’s own resampling unit, needs eight units and half its draws back, and is marked. Wilcoxon r, the matched-pairs rank-biserial r (paired and one-sample), Kruskal-Wallis ε² and η²_H, Kendall’s W, Kendall’s τ-b, Page’s ρ̄, Cohen’s g, Cochran’s average φ² and ξ have no analytic non-central distribution to invert; the iid-contribution variance behind a Wald interval misfires on a discrete lattice or bounded rank statistic, and a Fisher-z SE belongs to Pearson’s r under the bivariate normal, not to these constructions, so each is resampled and the interval taken at the percentiles, capturing the tie and continuity effects directly. The unit is the test’s: the signed deviations for the signed-rank measures – each replicate re-tested with wilcox.test on the branch the point estimate took, exact below 50 nonzero deviations, the continuity flag mirrored; observations within each group, sizes held fixed, for the Kruskal-Wallis measures and τ-b; subjects for W, ρ̄ and φ̄²; matched pairs for g. The count is the Bootstrap replications setting, seeded by Bootstrap seed, and the bounds are clamped to the measure’s range. The floor is eight units – eight nonzero deviations, eight cases, eight pairs, eight informative subjects, a smallest group of eight – below which the resamples are too few and too repetitive for the quantiles to mean anything, and the signed-rank measures also skip the interval at |r| = 1, where the variance vanishes into a zero-width interval; an interval is withheld when fewer than half the replicates return a finite value, and the tooltip on the b marker names the count that did. A replicate set that returns one value throughout – a Cohen’s g on a category pair where one transition direction has no counts to resample from – is at a boundary of the measure’s range, where a percentile bootstrap has no coverage, so the cell reports a marked blank rather than a spuriously exact [0.50, 0.50], and the forest plot, which needs finite bounds, leaves it out.

ξ is computed from the group data at the trim the test compared at, and at a balanced draw. Wilcox and Tian’s (2011) explanatory measure – the variance of the γ-trimmed group means over the winsorized variance of the pooled sample scaled by the standard normal’s cterm(γ), the formula behind WRS2’s yuen.effect and t1way.effect – is computed from the group data rather than read off WRS2’s object, whose own estimate is an unseeded Monte-Carlo median at unequal group sizes, needs a result object the heteroscedastic F omnibuses never produce, and pins its interval to an internal draw count that ignores the confidence level. γ is the run’s trim for Yuen and t1way and zero for Welch’s and Brown-Forsythe ANOVA, which compare untrimmed means, so ξ always describes the means the test compared; a two-group ξ past 1 takes WRS2’s winsorized-correlation substitute, and both the estimate and the bootstrap are clamped to [0, 1]. ξ is defined at a balanced design – the pooled winsorized variance’s between-group share peaks at equal sizes, so the raw unbalanced plug-in inflates it, 0.317 against 0.295 on a 20-versus-200 fixture – so at unequal sizes the estimate is WRS2’s median of the formula over draws of min(nⱼ) per group, pinned to the reproducibility seed so the magnitude reproduces run to run; the interval resamples those min-n draws with replacement, so each replicate is one draw where the estimate is a median over them, which is why it is marked conservative there – wider than the estimator’s own sampling distribution, erring toward coverage.

Standard errors are reported only where a closed-form SE exists on the measure’s own scale. The d family and the Algina-Keselman-Penfield robust d carry a delta-method SE, and rank-biserial r and Cliff’s δ Cliff’s consistent one, even where the displayed interval uses the more accurate asymmetric method – the non-central t, Cliff’s interval – so the SE is display-only and a d ± t·SE interval is neither the one shown nor a better one. The variance-explained and χ²-family measures, whose intervals are asymmetric non-central inversions, the point-biserial r and the common language ES, whose intervals live on the z and logit scales, the odds ratio, whose interval lives on the log scale, and every bootstrapped measure have no symmetric SE on the effect’s scale to begin with, so the column stays blank for them rather than carry a number the interval beside it would contradict.

Glass’ Δ reports Δ₁ and Δ₂ because the module has no control-group selector. Glass’ Δ standardizes the mean difference by one group’s SD rather than a pooled estimate, the right choice when the variances differ – the case Cohen’s d’s pooling breaks on – and by convention the control group’s; the module offers no way to say which group that is, so both are reported, Δ₁ over the first group’s SD and Δ₂ over the second’s. Under equal variances they coincide; the larger the SD ratio, the more they diverge, and in opposite directions, so reporting both keeps the reader honest about which standardizer is being read. Each gets its own magnitude label in the Interpretation column, since both are standardized effects on Cohen’s scale.

Cochran’s average φ² divides by the informative subjects and Cohen’s w (Q) by all of them; φ̄² borrows the variance-family benchmarks. φ̄² = Q/(N(k − 1)), the per-df form of w² for the Q statistic, bounded in [0, 1] and read as the fraction of maximum possible heterogeneity across conditions, where N counts the subjects whose response varied: a subject who answered the same way every time adds nothing to Q’s numerator or its denominator, so counting them would hold φ̄² below 1 and falsify that reading – it is the count the assumption panel screens Q on (Tate & Brown 1970). Cohen’s w (Q) keeps the full subject count, because w is conventionally √(χ²/n) on the sample the χ² ran on, and Q genuinely is computed from all of them; the two divide by different numbers, so each carries a superscript ⁿ naming its own denominator. No canonical threshold table exists for φ̄², so the module borrows the variance-family cutoffs – 0.01, 0.06, 0.14 – calibrated for η² and ω², a reasonable analogy for another bounded variance-explained-like ratio, and Cohen’s w (Q) sits on his 0.10 / 0.30 / 0.50 scale for anyone anchoring in the published thresholds; a φ̄² is reported with Q, N and k, since the same value carries different meaning at k = 3 and k = 8.

Stuart-Maxwell reports one Cohen’s g per discordant category pair. Cohen’s g measures marginal asymmetry as a deviation from no change: on a 2×2 table it is |b − c|/(2(b + c)), half the gap between the discordant counts scaled by the discordant pairs, not by N, bounded in [0, 0.5] with Cohen’s (1988) benchmarks of 0.05, 0.15 and 0.25. On a k×k table a single averaged g judged against those 2×2-calibrated thresholds would mislead, so the module reports gᵢⱼ = |nᵢⱼ − nⱼᵢ|/(2(nᵢⱼ + nⱼᵢ)) once per discordant category pair, each on its own footing and each with its own bootstrap interval over the matched pairs; a table with no discordant transitions anywhere reports a magnitude of 0, as the 2×2 branch does, rather than an empty table.

The trend tests’ companions restate J and L on a correlation scale, and ρ̄ is not Z/√N. J and L are on scales of their own, so each ordered-alternative test reports the measure conventionally paired with it. Kendall’s τ-b restates Jonckheere-Terpstra’s monotone association on a −1 to +1 scale, tie-corrected; J is a monotone function of it, so the two always agree in direction, and because τ-b is computed from the data it is available on the permutation branch too. Page’s mean within-block ρ is the average over blocks of the Spearman correlation between a subject’s observed ranks and the order the hypothesis predicts – tie-corrected, spanning −1 to +1 and comparable across designs with different numbers of conditions, where a Z/√N form would be capped at √((k − 1)/k), 0.816 at three conditions, so that “large” would mean something different at every k; a block whose ranks are all tied contributes 0, the same nothing it contributes to L. Both take bootstrap percentile intervals – τ-b resampling within group so the design’s group sizes stay fixed, ρ̄ resampling whole blocks.

Kendall’s W is labelled on Schmidt’s ladder, not on r’s benchmarks. W = χ²/(N(k − 1)) sits on the r scale, but it is agreement among the subjects about the conditions’ order, not a correlation, and Schmidt (1997) is the one benchmark table published for the coefficient itself: 0.1 very weak, 0.3 weak, 0.5 moderate, 0.7 strong and 0.9 unusually strong agreement, each value read as the floor of its band, so the lowest band runs to 0.3 and no W is negligible. The Reproducibility & agreement module grades its W through the same band function, so a W of 0.55 is moderate agreement on both cards where Cohen’s r cutoffs would call it strong and the ICC bands beside it moderate; the label carries the word agreement so the number is never read as an r, and Schmidt is cited only while the label is on screen.

Type III is the default on sum-to-zero contrasts, and a repeated measures factorial ANOVA without covariates reports Type III whichever was chosen. The control is offered to the tests fitted through car::Anova – factorial ANOVA, ANCOVA, repeated measures factorial ANOVA, mixed ANOVA and mixed MANOVA – and Type III is the default because it is order-invariant and what SPSS and most packages report, which makes it the reproducible choice; it is meaningful only under sum-to-zero contrasts, so the module sets contr.sum on every factor before the fit. Type II tests each main effect after the other main effects but not the interactions, has more power when no interaction is present and does not depend on the coding; on balanced data the two agree. Every other test fits Type III with no control, and ART-ANOVA partitions no sums of squares this way. In a repeated measures factorial ANOVA without covariates the between-subjects side is intercept-only, where car::Anova substitutes Type III tests regardless of the argument, so the card reports “Type III” whatever was picked rather than claim a partition that was not computed; with a covariate the two diverge and the choice takes effect. An empty between-subjects cell is refused under Type III and fitted under Type II, as the Choosing a test note says.

The within-subjects route is one analysis – omnibus, verdict, effect size, post-hoc gate and multiplicity pool – and falls back per term. The univariate route is the default because it exists at any sample size and is the safer one at small n; the multivariate route tests the within contrasts jointly by Pillai’s trace on the same car::Anova summary and needs residual degrees of freedom above the contrast count – below that the error matrix is singular and the test does not exist – so a term that cannot support it keeps its univariate F, uncorrected for sphericity since the ε machinery is not carried on that route, and is named in the diagnostics, which is what lets a card be multivariate for two effects and univariate for a third without misreading either. Under the multivariate route the sphericity tests, the ε columns and the corrected p-values are dropped, since they answer a question the route does not ask, and so are the sums of squares, since a mean square would not reproduce the approximate F beside it; between-subjects terms have no multivariate form and stay univariate; on a single-df within effect the two routes are numerically identical. The route’s one effect size is the partial η² recovered from Pillai’s approximate F, so the select narrows to it, and the runner applies the same narrowing to a programmatic configuration. A repeated measures ANOVA carrying a covariate is rewritten to a repeated measures MANOVA before it runs, which has no Pillai route on that path, so the control hides rather than let a selection be ignored.

The two-way decomposition is an SVD in base R; the joint one delegates to ca::mjca on the indicator matrix and adjusts the inertias alone. A two-way correspondence analysis is the singular value decomposition of the independence model’s standardized residuals over √n – their sum of squares is the total inertia, χ²/n, and their singular vectors the dimensions – so nothing a package adds is needed, and the result is checked against Greenacre’s smoke table and summary(ca::ca()) for every mass, inertia, coordinate, contribution and squared cosine; a margin emptied by complete-casing is dropped, and a table left with fewer than two rows or columns is refused. The joint analysis takes ca::mjca’s indicator solution, whose raw principal inertias are bounded by 1/Q and so read as negligible whatever the structure, and applies Greenacre’s (1993) adjustment to the inertias alone: mjca(lambda = "adjusted") also rescales the coordinates and divides them by inertias from a different matrix, which yields squared cosines above 1 that the package overwrites, whereas the indicator solution’s cosines sum to exactly 1 per category, so the coordinates, contributions and cosines come from it and only the inertia percentages are adjusted – on the dimensions above 1/Q, which is why the shares do not sum to 1 and the card says so. It needs three variables, since two make the two-way table, and more complete cases than variables; a variable with one realized level is dropped, and two variables that split the cases identically – a dimension of pure redundancy the map would lead with – are reported rather than refused.

Classification analysis is offered to the independent location and ordering tests alone, and never to a paired design. A t-test or Mann-Whitney asks whether the groups differ; ROC asks the dual question, how well the score discriminates them – the AUC is the Mann-Whitney U normalized to [0, 1] – and adds a cutoff with the metrics at it. The option is confined to the location and ordering tests because a pure dispersion test (the F-test) or a whole-distribution test (Kolmogorov-Smirnov, Jensen-Shannon) can report a large and entirely correct difference next to an AUC near 0.5 – groups with the same centre and different spreads are genuinely hard to tell apart with one threshold – which would read as a contradiction inside one card. A paired or repeated-measures design is excluded because the AUC’s machinery assumes independent observations, so the standard interval is wrong on pre/post data, and “can this score discriminate pre from post within the same subject” rarely matches the question asked; discrimination on paired data is difference scores against an external label in an independent run. The fit is pROC::roc with direction “<” on the encoded factor’s level order, needs exactly two groups and at least four scored cases, and a variable whose AUC comes out below 0.5 has its levels reversed and is re-fitted, on pROC’s rank-based check rather than a mean-based pre-orientation, which is unreliable for bimodal or heavy-tailed scores – so that the predicted group is always the second level and every AUC is at least 0.5, except while AUCs are compared.

The threshold rules are pROC’s coords(best) with ties broken toward the corner, and the cost ratio is strictly above 1 with both directions reported. Youden’s J and the cost-weighted rule are best.method = "youden" with best.weights of (1, 0.5) and (cost, prevalence); closest to (0, 1) is closest.topleft. coords() can return several equally optimal rows in incidental order, so a tie is broken deterministically by distance to the (0, 1) corner. The cost rule reports two rows – the ratio with the missed positive as the costlier error, its reciprocal with the false alarm – because a ratio can be meant either way round; the field therefore accepts values strictly above 1, since any asymmetry below 1 is its own reciprocal above 1 and nothing is lost, while exactly 1 asserts no asymmetry at all: both rows would compute the same weights and land as identical duplicates carrying contradictory “costlier error” labels, duplicating what Youden’s J already reports. An out-of-range value is refused with a message rather than silently replaced.

The prevalence override travels with a named level and recomputes PPV, NPV and accuracy from sensitivity and specificity; the metrics’ intervals re-choose the cutoff in every resample. The runner picks the predicted group from the data per variable and per group pair, so an unbound share would mean P(A) = p on one row and P(B) = p on the next; the override is therefore a statement about one named level – one select per factor the run fits, as for the control group, since a batch run fits each grouping variable in turn and a level of one is not a level of the next – defaulting to the last level, the one pROC treats as the cases in an unflipped fit, and mirrored to 1 − p when the fit left the other level as the predicted one; a level that names no level of the fitted factor leaves the sample rate standing, and each card says which prevalence reached it. coords()'s own PPV, NPV and accuracy always use the raw sample prevalence and ignore the weights, so the three are derived from the sensitivity and specificity at the effective prevalence – identical to pROC’s when it is the sample rate – which is also how the override moves the cost-weighted cutoff, whose weights carry it. Because the cutoff is chosen in-sample, the metrics at it are optimistic point estimates; the interval re-selects the cutoff inside each stratified resample, class sizes held at the observed ones, so the selection variability is in it – derived by hand rather than through pROC::ci.coords, which fixes the predictive values at the sample prevalence and would contradict the override, and with one resample serving every rule, so the cost rule costs one loop, not two; the count is the Bootstrap replications setting.

Comparing AUCs filters the fits listwise and pins one orientation; the p-values are pooled as one family. Paired DeLong assumes every curve is fitted on the same subjects, so when AUCs are compared the rows missing on the grouping factor or on any selected dependent variable are dropped from every fit up front – otherwise pROC’s pairing auto-detection is ambiguous where the variables differ on which rows are missing – and the per-AUC N is that shared subset; a single fit, or several with no comparison, complete-cases each variable on its own, so an NA in one does not shrink the others. The per-variable flip is skipped while comparing, since paired DeLong rejects curves whose case/control partitions differ, so a variable that discriminates in reverse legitimately shows an AUC below 0.5, and its operating point is taken from the mirrored decision rule – lower scores favour the predicted group – so that the cutoff is usable while the AUC and its interval stay on the shared orientation the test needs. roc.test is called with paired = TRUE for every pair of variables and leaves the p-values raw; they are adjusted downstream as one family across every variable pair, group pair and batch factor, so that the pairwise expansion and batch mode do not leave several small pools that under-correct, and kept apart from the omnibus pool, an AUC comparison being a different hypothesis from a group difference. The AUC’s own interval is pROC::ci.auc – DeLong’s closed form, or the bootstrap at the Bootstrap replications setting.

The direction phrase reads one summary statistic per family, computed whether or not its box is checked. The Interpretation column’s “Group A > Group B” needs a group statistic to name a winner – the mean for the location tests, the median for the rank-based ones, which compare location rather than means and can disagree with the mean on heavy-tailed or tied data, and the standard deviation for the dispersion tests, which reject on spread – so buildCalcFlags forces that one statistic’s computation while table and CSV visibility stay gated on the user’s picks; without it the direction line would silently disappear when the box was off. The confidence-interval box also gates the difference’s own interval column and the intervals of the pairwise tables, since they are read at the same level.

Morey’s interval stays the default bar, and a within-subject cell overrides a CI or SE request. Both within-subject intervals remove between-subject variance, which is what makes them the right bars for comparing repeated levels; they differ in the rule that can be read off overlap. The Cousineau-Morey interval (Morey 2008) is the standard construction and what the tables print, but two of its bars just touching does not correspond to p ≈ .05, so overlap is not a test; the difference-adjusted interval (Baguley 2012) is that same interval scaled by √2/2, narrower, precisely so that the rule holds – two bars that do not overlap mark a significant difference. Morey’s stays the default because it is what the surrounding literature and other packages draw, and switching silently would rewrite every figure the module has already produced; the caption under each plot names which is drawn. A cell whose stats carry the Morey interval draws it whatever a CI or SE request said, since its SE is the subject-normalized one and has no between-subject reading, while SD survives as asked, being the plain spread of the values; a difference-adjusted request with no such cells drops to the ordinary CI rather than shrink a between-subject one, which is why the option is offered only where a within factor exists and the SE option is withdrawn on the purely within-subjects path.

Checking assumptions #

Normality is tested on what the selected family assumes normal, never on the pooled variable. A dependent variable pooled across groups is a mixture of the groups’ distributions and fails a normality test exactly when there is a group difference, so the check follows the model: shapiro.test per group in an independent design, per cell of the between crossing in a factorial one, and on the within-pair differences in a two-condition dependent design, which is the quantity the paired t-test is computed on. With three or more conditions the assumption is on the residuals of lm(dv ~ subject + condition) over the subjects complete in every condition, with condition × covariate slope terms added for a repeated measures ANCOVA so the residual space matches the fit the verdict gates; with a covariate in an independent design it is on the residuals of lm(dv ~ grouping + covariates), since when covariate spread differs between groups the raw group values are non-normal and heteroscedastic under a perfectly well-behaved model, and the same residuals serve the Levene rows. A mixed design tests every between × within cell over the subjects complete across the conditions – the listwise sample its lm fits – and past 50 cells, or when the per-cell pass errors, degrades to one Shapiro-Wilk per factor level on subject means, since the raw long rows would pseudo-replicate each subject across the levels of a factor and inflate every n; the cap keeps the rendered matrix usable, an 81-cell 3 × 3 × 3 × 3 crossing being mostly cells of fewer than three. Per-cell rows carry their values centred on the cell mean, so one variable’s rows concatenate into the pooled within-cell residual the Q-Q plot draws and the outlier test reads. Below 3 values shapiro.test cannot run and above 5,000 it is withheld, because at that n it rejects departures too small to matter; skewness and excess kurtosis are computed for every cell regardless, so an untested cell still reads.

The outlier test is Bonferroni’s on externally studentized residuals, and it never demotes. A fixed |z| > 3 cutoff flags a fixed fraction of every clean normal sample, so its count grows with n and says nothing about the sample. Each residual is instead divided by the model’s error with its own observation deleted – the leverage 1/n for a cell-means fit, the fit’s own hatvalues and df.residual where a model’s residuals were tested – referred to t(df − 1), and its two-sided p multiplied by the number of observations in the variable’s pool, so the flag rate stays at the assumption test significance level whatever the sample size. It needs df ≥ 2 (the deletion costs one more), skips a slot at leverage 1, which leaves no residual to test, and is withheld on the per-factor fallback, whose between and within rows are marginals over the same subjects and would count every observation twice. A flag is never a demotion, since an extreme observation is a case to look at rather than a violated assumption and what to do with it is the user’s call; the location tests take a caveat, and the F-test and Bartlett’s a sharper one steering to Fligner-Killeen, because a variance follows an extreme value by its square.

The symmetry check flips signs rather than trusting a normal-theory variance for the skewness. The Wilcoxon signed-rank test assumes the differences – or the values about μ₀ – are symmetric, not normal, and Shapiro-Wilk cannot separate the two: a symmetric heavy-tailed sample fails normality and is still fine for the signed-rank test. D’Agostino’s (1970) test of g₁ would do, but it rests on Var(g₁) = 6/n, a normal-theory value a symmetric heavy-tailed sample exceeds many times over – 63/n for the Laplace – so it rejects symmetric data for their tails. The null is built instead by flipping the signs of the deviations from the median at random, which is valid under any symmetric law and is exactly the null “the differences are symmetric”: 2,000 replicates under a pinned seed, the p in the never-zero form (1 + #{|g₁*| ≥ |g₁|}) / (B + 1) (Phipson and Smyth 2010). Below n = 6 no sign pattern is extreme enough to reject at a conventional α – the smallest two-sided p is 2/2ⁿ – so those rows read n < 6 rather than passing. A failure steers to the sign test that fits the design, which reads only directions and assumes nothing about shape, and never demotes, since the signed-rank test’s error rate moves gently under mild asymmetry.

Mardia’s test runs per group with the small-sample corrections, on the model the multivariate test fits. The joint normality MANOVA assumes is within each group, so the test runs per between cell and fails on any group’s skewness or kurtosis rejecting; a covariate moves it to the residuals of lm(cbind(dvs) ~ grouping + covariates), and a within-subjects design reshapes to one row per subject with a column per variable × condition, since the rows Mardia assumes independent are subjects, not observations – one group for a pure repeated measures design, one per between cell for a mixed one. The statistics use the biased n-divisor covariance the reference distributions assume, Mardia’s (1974) small-sample factor on the skewness χ² and the finite-sample centring p(p + 2)(n − 1)/(n + 1) on the kurtosis z, matching MVN::mvn’s corrected statistics at small n and the classical ones as n grows; the kurtosis p is two-sided, since platykurtic departures break the assumption as much as heavy tails. A group needs more complete cases than variables, or its covariance is singular – a near-collinear pair fails chol() the same way and is reported likewise – and a run in which no group could be assessed reports not assessed rather than a pass the recommender would read as met.

Levene’s test is the median-centred Brown-Forsythe form, on the sample the test fits, and a mixed design gets it twice. car::leveneTest defaults to center = median, the modification robust to the skew and discreteness the location tests’ data often carry; rows with a missing value are dropped first, since the function has no na.rm and a group whose median goes NA is dropped wholesale. With a covariate the rows describe the residuals of the covariate-adjusted model, for the reason the normality note gives. A mixed design cannot run it on the long frame – each subject contributes one row per condition and df2 inflates by the repeat count – so the between-groups row runs on each subject’s mean over the conditions, where the rows are independent, and a second family runs across the between groups within each condition on the subjects complete across every condition, the listwise sample the mixed ANOVA fits; it is the second that the mixed ANOVA’s F-tests rest on, and both demote the same key under one balance gate. Levene is the gatekeeper because it is an ANOVA on absolute deviations and so robust to non-normality, which Bartlett’s and the variance F-test are not – those two are the module’s tests of spread as a hypothesis, demoted on any normality failure and steered to Fligner-Killeen.

A per-cell family demotes only once its failures outrun chance. A wide mixed or factorial design runs one normality test per cell and one Levene per condition × variable – fifty of either in a single check – and at α = .05 a family of fifty throws two or three failures under a perfect null, so “any failure demotes” would read “always demotes”. The bound is the largest failure count a family of m independent tests still reaches 95% of the time under Binomial(m, α), and a family is treated as violating the assumption only past it; at m ≤ 4 the bound is forced to 0, since the binomial bound is already 1 at m = 2 and would excuse the single failure that is a narrow family’s whole evidence. For normality the family is the rows without central-limit cover alone – a covered row’s rejection can never demote, so counting it would inflate the bound until a genuine small-cell failure was dismissed – and the covering n is the subject count for a within-subjects model’s residuals, the smallest cell for a covariate-adjusted model’s. The per-cell rows are always shown raw and the summary rows disclose the bound in the same breath as the count, so a visible failure count never reads as contradicting the recommendation beside it.

The spread of the cell standard deviations carries no threshold, and is shown only where an ART test is offered. ARTool fits (1|subject) as the whole within-subject covariance, which assumes one residual variance across every between × within cell and corrects no departure: under a true null a cell-SD ratio of 3 took the omnibus from .0475 to .275, and the per-condition Levene cannot stand in for the check, since it compares the between groups inside a condition and never one within level against another. The table therefore reports the smallest and largest cell SD and their ratio on exactly the sample the ART fit uses. No verdict is attached because none is honest: the ratio’s null distribution runs 1.40–2.14 for a homoscedastic design at 15 per cell, and where ART actually breaks depends on which axis the spread lies on – a population ratio of 2 across within levels, 3 across between groups – so no one cutoff on the printed number serves. It is computed for every dependent and mixed design but drawn only when an ART key is among the recommendations, since a two-condition paired design carries one in the not-recommended list and the note would then explain the assumption of a test the user cannot pick; a cell with no variance keeps its row rather than vanishing.

Mauchly’s test comes off the omnibus’s own wide fit, per within-subjects term, and its verdict is a caveat. The sphericity fit is one R helper shared with the repeated measures ANOVA, so the panel and the results table cannot disagree: car::Anova on the wide subject × cell response with the intra-subject design, between-adjusted when a mixed design carries grouping factors, covariate-adjusted when the fit will be, and one Mauchly row per within term – main effects and interactions – since car reports each contrast on its own and omits a one-df contrast, which has no sphericity to test. An incomplete within crossing is fitted on the combined factor over the observed cells, as the analysis fits it, or the panel would report “could not be verified” for the very model whose corrections the results table prints. Fewer than three within cells is not a limitation but sphericity by construction, reported as not applicable; too few complete within cells is one, since the fit only runs on a design with three or more. The n column is the subjects the fit kept, disclosed because Mauchly’s power hinges on it and the fit drops incomplete subjects silently. A violation is a caveat rather than a demotion because the Greenhouse-Geisser and Huynh-Feldt corrected p-values are already the columns the results table reads its verdict from, so the test self-corrects the way the Welch tests do under unequal variances; and the test is left at the assumption test significance level with both corrections shown, since it is underpowered at small n and over-rejects at large n, so a reader can act on the corrected p either way.

Box’s M is read at a fixed α = 0.001 and demotes only an unbalanced design. The test is sensitive to non-normality and over-rejects at large n in every design, so its threshold is the textbook 0.001 (Tabachnick and Fidell), deliberately independent of the assumption test significance level. What a rejection costs depends on balance: with roughly equal cells – the largest no more than 1.5× the smallest, the same gate the per-condition Levene uses – Pillai’s trace absorbs covariance heterogeneity and the pooled family stays recommended with a caveat, while unequal cells demote it, MANOVA and MANCOVA steering to Johansen’s, which weights each group by its own matrix and holds nominal size where Pillai’s runs to .406 or .002 depending on which group is the over-dispersed one. The statistic is Box (1949)'s with its χ² correction on p(p + 1)(g − 1)/2 degrees of freedom, computed on the residual matrix of lm(cbind(dvs) ~ grouping + covariates) for a MANCOVA – as biotools::boxM would report it, without deducting the covariate degrees of freedom – and needs more complete cases than variables in every group, since a singular group matrix has no log-determinant. A mixed design reshapes to one row per subject with a column per variable × condition and tests those matrices across the between groups: multi-sample sphericity, which the univariate mixed ANOVA rests on too, so a single dependent variable still has a response of two or more columns and the verdict reaches mixed ANOVA under the same balance split; it needs at least five complete subjects per group and more than the response has columns. The energy test is demoted on any rejection, ungated by balance, because unequal covariance is not a nuisance for it but a difference its null rejects on – softened to a caveat once multivariate normality has failed too, when a normal-theory test of the covariances establishes nothing and a demotion would leave no multivariate test recommended.

Collinearity is read on the error covariance at |r| > 0.90. Pooling the groups adds their mean separation along both dependent variables to r, so a pair the model never sees as collinear can clear the cutoff; the correlation is taken on the residuals of lm(dv ~ grouping + covariates) for a between design and of the subject + condition fit for a within or mixed one, which is the covariance the multivariate test inverts. The cutoff is one constant shared with the R side and the correlation module, so the three cannot drift. A flag demotes the tests that invert a pooled matrix – the MANOVA family and Johansen’s, whose per-group inversion is worse off – and not the energy test, which computes distances; the same check within the covariate set is a caveat rather than a demotion, since the model still fits and which covariate to drop is a judgement the panel cannot make.

The covariate checks are the interaction, a quadratic term and a one-way ANOVA, on one row per subject. Homogeneity of slopes is the grouping × covariate term of lm(dv ~ grouping * covariate) per dependent variable × covariate, the standard test; linearity adds I(covariate²) to lm(dv ~ grouping + covariate) and tests it, so a curved within-group relationship fails; independence is aov(covariate ~ grouping), the covariate itself compared across the groups. A within-subjects design defines the factor as the condition and carries subject as a fixed blocking effect: slopes are the condition × covariate term, linearity is tested at the subject level, and independence is not tested at all, since a covariate that takes one value per subject cannot vary with a within factor. A mixed design fits the between-side diagnostics on one row per subject – the covariate and the dependent variable averaged over the subjects complete in every cell – because the long frame would count every subject once per condition and inflate the residual degrees of freedom, and adds the condition × covariate slope rows from the within pass, axis-tagged, since the results-side region of significance is post-hoc and nothing else checks that side. A slopes failure names the follow-up the card runs rather than the bare violation, because the demotion is about the single adjusted difference, not about the design having no answer.

Cochran’s rule is applied where the test is fitted: per table, per stratum, or on the hypothesised counts. The rule – no expected count below 1 and at most a fifth of the cells below 5 – is one R helper the chi-square runner and both panel rows interpolate, so the inline warning and the panel verdict cannot drift. A contingency table drops its all-zero margins first, so an unobserved level adds no spurious expected-0 cell, and the expected counts come straight from the margins, which still yields a verdict where chisq.test would error. With a stratifying variable the panel mirrors the Cochran-Mantel-Haenszel runner: strata of one observation are dropped, the expected counts are pooled over the per-stratum tables, and on a 2 × 2 × K table the Mantel-Fleiss criterion – the first cell’s expected count summed over strata at least 5 from both bounds of its support – decides instead, the note warning about the CMH approximation because Fisher’s exact test is not the remedy there. The goodness-of-fit row derives expected = N × pᵢ from the entered weights, a uniform null while nothing has been entered – chisq.test’s own default – and reports an entered hypothesis with fewer than two categories or a non-positive weight as untested rather than swapping in the uniform, which would green-light a hypothesis never specified; the run refuses the same weights. Nothing on these rows demotes, since the categorical tests are excluded from the recommendation lists – an unassigned categorical variable is auto-treated as a dependent variable, which would make their verdicts misleading – and the goodness-of-fit runner switches to a simulated p on its own.

The paired categorical rows count what their test uses – discordant pairs at 25, informative subjects by Tate and Brown. McNemar’s and Stuart-Maxwell’s statistics are built on the off-diagonal pairs alone, so the criterion is their count, at the 25 below which the McNemar wrapper switches to the exact binomial; on three or more conditions the two-sample tests expand to one test per condition pair, so every pair is counted and the weakest reported. The note differs by category count because the remedies do: with two categories the exact test takes over automatically, with more there is no exact Stuart-Maxwell and only a reliability warning is honest. Cochran’s Q gains nothing from a subject whose response never changes, so its effective sample is the informative subjects, adequate for the χ² approximation when N ≥ 4 and N × k > 24 (Tate and Brown 1970), and only a binary variable qualifies. Which criterion applies follows the selected test rather than the level count, since McNemar on a three-level factor is still a discordant-pairs question.

A demotion is a violated assumption the sample does not cover; everything else is a caveat. The parametric location tests are demoted for non-normality only where a failing cell is under the small-sample advisory floor – 30 by default, a setting, so the caveat names the bar rather than a number – and the covering n is the unit the test’s robustness rests on: subjects for a within-subjects model’s residuals, the smallest cell for a covariate-adjusted model’s. Bartlett’s and the variance F-test are demoted on any normality failure, since a variance has no central-limit rescue, and Welch’s tests are absent from the Levene list because they assume no homogeneity – they are what the demotion steers to. Sphericity, extreme observations, an unverified check and a balanced design’s Box’s M or by-condition Levene failure are caveats, for the reasons their notes give; the never-demoted robust options carry an apology while the assumption they relax holds – Fligner-Killeen under normality, the trimmed-means tests unless normality and homogeneity have both failed, Johansen’s while Box’s M passes or the cells are balanced – so they are not picked by default over a standard test whose assumption is met, and the sign tests and both ART tests carry theirs always. A test the selection cannot support is listed as not recommended with the requirement it lacks, mirroring the run’s own pre-flight so the panel never recommends a test the run then refuses, and “not assessed” while a role is empty rather than a recommendation made blindly. The categorical tests are excluded from both lists.

The Q-Q plot pools the rows and thins in rank order. One plot per dependent variable on the residuals the normality rows were tested on, joined from the per-cell rows that carry their values centred on the cell mean, so the cost is one plot per variable rather than one per cell, and a note says the pooling can let departures in opposite directions cancel. Fewer than 5 values draw nothing, the gate the regression and time-series modules put on theirs. Past 5,000 residuals the plot takes every k-th value in rank order: a Q-Q plot is a function of the order statistics, so that is the same curve at lower resolution, where a random subsample would be a different one. The per-factor fallback draws none, since its between and within rows are marginals over the same subjects. The note beside the plots reads the sample-size-robust part of the table out loud – W as the plot’s own straightness, skewness its curvature, excess kurtosis its tail weight – and sends the reader to the Normality analysis view for the full battery rather than to Distribution analysis, which plots the raw marginal, a mixture across groups that looks non-normal precisely when a real effect exists.

Reading results #

The intervals on a one-factor card stay at the nominal level while its p-values are adjusted. The run-wide adjustment writes p-values alone – every interval comes back from R at the configured confidence level, and no interval is dual to a step-down or false-discovery procedure – so an adjusted p can read non-significant beside an interval that excludes zero. The card says so in a note under the table whenever an adjustment is on and any interval is drawn – a per-group interval, the difference’s, an effect size’s – rather than widening the intervals to a Bonferroni level the adjustment may not be; the post-hoc contrasts are the exception, since emmeans widens its own intervals under its adjust=, which is why they are headed simultaneous CI.

The verdict is worded for the null the family states, and the direction line reads the test’s own estimate. Significant difference would misreport a dispersion test, a whole-distribution test, a goodness-of-fit or a proportion test, whose nulls are equal variances, one distribution, an expected distribution and π₀, so each family has its own wording, and the one-tailed forms exist only for the families whose direction control is offered – the location tests, the F-test, Fisher’s 2 × 2 and the proportion test. The direction line takes the sign of the difference estimate where the test reports one, because a rank test’s Hodges-Lehmann shift and the two marginal medians can disagree in sign, which would contradict a one-tailed verdict beside it; where there is none it orders the groups by the family’s summary statistic – the mean, the median for the rank and whole-distribution tests, the SD for the dispersion tests – and names that quantity for the families whose descriptives print another, since an unlabelled A > B beside two means reads as a claim about the means. The Cochran-Mantel-Haenszel row prints no direction line at all: its frequency table is the pooled one, whose marginal proportion can point opposite to the common odds ratio beside it – Simpson’s paradox – so the odds ratio and its interval carry the direction.

The repeated-measures ω²p is the noncentrality plug-in its interval inverts. ${\omega }_p^2=\frac{d{f}_1({\text{MS}}_1-{\text{MS}}_e)}{d{f}_1{\text{MS}}_1+(d{f}_2+1){\text{MS}}_e}$, the λ̂ plug-in into Steiger’s (2004) proportion-of-variance transform, so that the estimate and the non-central F interval around it sit on one scale; Olejnik and Algina (2003), effectsize and SPSS put the total number of observations where df₂ + 1 stands and report a slightly smaller value, which the note under the table names rather than let the two be reconciled as an error.

Page’s L takes its exact null distribution below 16 conditions and 21 blocks, and only without ties. Page (1963) tabulated the exact distribution of L per block, and DescTools::PageTest convolves it across blocks for k < 16; the table assumes untied within-block ranks, so a tied block keeps the normal approximation, whose variance is computed from the tie-averaged rank sums – the closed form $n{k}^2(k+1)(k^2-1)/144$ ignores the ties and leaves p anti-conservative. The convolution runs by FFT, whose round-off swamps the tail p well before 40 blocks and turns it negative outright, so the exact value is taken only up to 20 blocks, where it stays a probability, and the approximation – accurate by then – above. The two differ materially in exactly the small-k regime the test is built for, which is why the exact branch is taken wherever it is sound.

The one-factor card’s table of counts is drawn for a numeric outcome at 20 or fewer distinct values. The Joint frequency table option exists for the categorical tests, but a rating on a short scale is the same object, and above 20 distinct values a table of counts degenerates to one row per observation – unreadable and slow to build – so the independent and paired families share one ceiling, which drifted apart while each runner carried its own.

Standardized residuals are marked at a critical |z| Bonferroni-adjusted over the cells. A standardized residual is asymptotically N(0, 1), so the cell-wise follow-up to a chi-square or goodness-of-fit omnibus is |z| against a critical value, and a bare 1.96 flags one cell in twenty on a table where nothing departs; the bound is Bonferroni’s over the number of cells examined, so a marked cell is one extreme enough to survive scanning the whole table. It is chisq.test’s $stdres and not its $residuals: the Pearson residual has variance below 1 and under-flags at any cutoff read as a z. The category proportion plot marks its bars from the same function, so the chart and the table cannot disagree, and for the goodness-of-fit test this is the only post-hoc there is.

The TOST verdict is Lakens’s four-way reading, and the equivalence decision follows the adjusted p. The forced two-sided test and the two one-sided tests are independent decisions, so crossing them (Lakens 2017) separates equivalent from statistically different but equivalent – a real difference too small to matter – and different, equivalence not established from inconclusive, where neither test rejects and the sample says nothing either way; the two-way reading is kept only where the two-sided p could not be computed. What enters the run-wide adjustment is the decision p – p (equiv) or p (MET) – and the verdict reads the adjusted value, since a two-sided p adjusted in its place would leave the equivalence claim uncorrected for the family it was made in; in addition mode both adjusted p-values are printed, because the four-way verdict reads both.

Stuart-Maxwell inverts its covariance matrix by singular-value decomposition and reports the effective rank as df. The statistic is $d^{\prime }{S}^{-1}d$ over the k − 1 marginal differences, and S is singular whenever a category is empty or two categories are perfectly redundant – exactly the sparse paired tables the test meets. A plain solve() would fail with a raw R error, and a tryCatch around it would hide which of two cases occurred, so the runner decomposes S once: when every singular value is below tolerance the test is undefined and the card reports NA with a note, rather than the χ² = 0, p = 1 a pseudo-inverse would silently produce; when some are dropped the pseudo-inverse over the surviving directions is used, the df reduced to that rank so the p stays calibrated, and a note calls the result approximate, since collapsing redundant categorical structure is a decision the reader should know was made.

The pairwise summary names the family the adjustment ran over, because the two flat pairwise shapes are pooled at different scopes. An automatic expansion has no omnibus row – each pair is a test in its own right, so its p-values join the run-wide family with every other test in the analysis, the Choosing a test note’s rule; an opt-in breakdown of an omnibus decomposes a row that is already in that family, so its pairs are adjusted within the variable’s own set – the structure the R-side post-hoc tables already have – and the omnibus row stays in the pool, which is why switching the breakdown on tightens nothing else in the run. Both shapes go through the same engine and the same selector, and the flat tables carry no Method: line of their own, so the summary box above them names the method and the scope rather than leave the correction implicit; a family of one comparison is left raw, since every adjustment collapses to the raw p there.

A matrix cell’s mirrored orientation is derived per statistic, and a one-sided equivalence p is a second fit. The lower triangle reads row − column uniformly, so the orientation the runner did not fit is derived from the one it did: a location statistic negates; var.test’s F is the variance ratio itself and reciprocates, the one dispersion statistic that does; a rank-sum count complements against its maximum – U′ = n₁n₂ − U, V′ = maxV − V, the sign count against the non-tied n – since negating a count would print a wrong sign; and a symmetric statistic – a categorical χ², an information-theoretic divergence, D, T.AD – is kept, being orientation-invariant. The effect size flips on its own scale, keyed off its null value and family – a ratio reciprocates, a probability of superiority complements to 1 − v, a signed d, r or δ negates, a symmetric magnitude such as Cramér’s V or ξ stays – with its interval mapped and swapped, every transform being monotone decreasing, and Glass’ Δ₁ and Δ₂ trade places with their SE and interval, since reversing the pair also swaps the standardizer. A one-sided equivalence p is the one field that is not a transform of its own value: “group 1 is non-inferior to group 2” and its converse are different hypotheses against opposite shifted nulls, so the runner fits both orientations and the mirrored cell shows the second, adjusted as its own parallel family, while TOST and the minimal-effect test, symmetric in Δ, pass through.

The post-hoc contrasts of a within-bearing design each carry their own error term, and the note above the block says which. The repeated measures, repeated measures factorial and mixed families – and the multivariate designs built on them – contrast every term on the multivariate lm, whose unstructured within covariance gives each contrast the error of the cells it involves rather than the split-plot pooled MSE the omnibus row is computed from, so a within contrast stays valid when sphericity fails; the price is fewer degrees of freedom per contrast, and a contrast that need not agree with the omnibus verdict. A between-only contrast is unaffected: the multivariate fit reproduces the aovlist’s pooled between SE and df exactly, balanced or not. The ART families pool instead, art.con handing every pair of a term one SE and one df, which redistributes the error rate rather than inflating it – the ART notes – and the factorial card sets neither note, since it pools too but with no within factor has no sphericity for the pooling to assume. The two are R-side flags on the post-hoc payload rather than test keys, because the renderers are shared across the families.

The Brier score is Platt-calibrated in-sample, and blank where the fit is unidentified. A Brier score needs probabilities and a raw dependent variable is not one, so the score is calibrated with a one-predictor logistic fit (Platt 1999) and the fitted probabilities are scored against the predicted-group indicator; the fit carries the orientation, so a reverse-discriminating variable under a shared orientation needs no separate mirroring. The calibration is fitted on the same cases it scores, so the number summarises fit rather than out-of-sample forecast skill, and the card’s legend says so; when every score is the same the slope is unidentified and both cells are left blank rather than reported as 0. The skill score rescales against the base-rate constant predictor, whose Brier is p(1 − p), the reference that keeps it comparable across variables with different group balance.

The factorial card prints its adjusted means always, from the same sum-to-zero fit as the omnibus. Without covariates they are the equal-weighted marginal means – each level’s the average of its cell means whatever the cells’ sizes – which is the quantity a Type III main-effect null is about, and the only block on the card that matches the test above it: the descriptives beneath are sample-size-weighted, so on an unbalanced design the two differ and a reader who saw the raw means alone would reconcile the verdict against the wrong quantity. They are therefore computed whether or not post-hoc tests are on – the mixed card, whose marginal means come from a separate wide-format fit, waits for something to ask (the adjusted-means note) – with the cell crossing first and the per-factor marginals under it; with covariates present the same block is covariate-adjusted, emmeans pinned to the fitted frame so every mean is evaluated where the held-at line says it was.

η², ω² and ε² on a source table divide by the corrected total, re-based from effectsize’s denominator. effectsize’s non-partial measures divide by the sum of the effect sums of squares plus the error’s – plus the error mean square for ω² – and under Type III with imbalance the variance the terms share is credited to no term, so that sum stops matching the mean-centred total: usually short of it, the other way in about one sample in eight. The corrected total is the one denominator a reader can check from the foot of the table, so the estimate and its non-central-F interval are re-based onto it – one positive factor, the numerator unchanged, so the interval maps with the point estimate – and on a balanced design the two coincide and nothing moves; partial η² is hand-rolled from the SS decomposition with the same Steiger (2004) interval and is unaffected, being partial by construction. The note under the table names the denominator so that a package keeping the other one can be reconciled, and a measure the package could not compute falls back to partial η², relabelled and disclosed in the diagnostics rather than reported under the chosen name.

The adjusted means come from the multivariate lm, on its own fitted rows, and the mixed card prints them only when something asks for them. emmeans on an aov() + Error() fit mis-separates a covariate from the intercept stratum and biases the between means, so the mixed families’ marginal means come from the wide-format lm the sphericity tests use, with data = model$model: lm() drops incomplete subjects listwise, and a covariate held at the mean of rows the model never saw shifts every adjusted mean and widens its interval – which is also why the ANCOVA card’s raw means, N and SD are read off the fitted rows rather than the loaded data. Without a covariate they are the equal-weighted marginal means a Type III main-effect null is about under imbalance, so the mixed card computes them whenever post-hoc tests or a covariate call for a contrast, Mixed MANOVA always, and MANOVA never, since there they would be the group means the descriptives already print. The value each covariate is held at is printed under the heading because an adjusted mean is a prediction at that value and nothing else – SPSS and SAS print the same line – and on a card whose per-variable fits kept different rows the line names no value, since each variable’s is its own.

Every within-subjects row is reported on its ε-corrected p, chosen by Girden’s rule, and the multiplicity pool takes the same p. The verdict, its tag and the pooled p read the Greenhouse-Geisser p when ε̂ (GG) is below 0.75 and the Huynh-Feldt p from 0.75 up (Girden 1992) – the GG estimate runs low near sphericity and the HF estimate is the less biased one there – on every within row, not only where Mauchly’s test rejects, for the reason the assumptions note gives: where sphericity holds ε̂ sits near 1 and the correction barely moves the p, and gating on Mauchly would track that test’s power rather than the data. The pool and the displayed verdict go through one helper, so they cannot part; between-subjects rows carry no ε and stand on their raw p, ART has no sphericity notion and is never tagged, and the MANOVA family’s univariate follow-ups on a within design read their verdict the same way.

ART is fitted as a random-intercept model, checked by ARTool’s own criterion, and tested on Kenward-Roger denominators. ARTool::art with (1 | subject) routes the fit through lmer, which handles any between × within crossing where the aov() / Error() branch needs nested strata; anova.art tests it through car::Anova with pbkrtest’s Kenward-Roger denominator df, hence the fractional error df on every row. The fit is accepted on ARTool’s two conditions – the aligned responses sum to zero, and an ANOVA on every aligned response not of interest returns an F within all.equal’s 1.5 × 10⁻⁸ of 0 – and a failure is disclosed in the diagnostics with the F it found, since the aligned-rank F-values are uninterpretable then; an empty between × within cell and unequal between-cell sizes are the usual causes, and a within crossing collapsed to one factor leaves nothing to align against, so the check does not run there and the card says so. A singular random-effect covariance – too few subjects per between cell, collinear factors – makes the Kenward-Roger step fail, which the runner catches and reports by its cause rather than as the package’s message. A covariate is dropped with a diagnostic rather than refused, since the alignment is defined for categorical factors alone. Partial η² is the one effect size because the aligned response has no variance partition on the outcome’s scale: it is recovered from F and the two df, and its interval inverts the non-central F, as the parametric family’s does.

ART contrasts come from art.con, one term at a time, on one pooled error. Plain emmeans on the aligned model would contrast the wrong response: art.con re-aligns and re-ranks for each term (ART-C), a main effect on its own aligned response and a two-way interaction as difference-of-differences contrasts (interaction = TRUE), the only contrast alignment supports – a simple effect at a fixed level of another factor reads exactly the structure the alignment removed, so none is offered, and three-way terms keep their omnibus row. Every pair of a term takes one SE and one df from the pooled fit, which redistributes the error rate rather than inflating it: measured under a null with equal cell variances but AR(1)-correlated conditions, one pair ran at .0033 against another’s .1067. The note under every block says so, and the spread of the cell standard deviations under the source table is what to read the set against.

A multivariate effect’s η²p is recovered from Pillai’s approximate F, and its adjusted p is pooled once. The four statistics test one hypothesis, so the global adjustment takes the first valid row’s p – Pillai’s – and the card prints the adjusted value and its verdict on that row alone rather than four times; the covariate tests stay out of the pool as nuisance terms. The effect size is F·df₁ / (F·df₁ + df₂) from Pillai’s row with a non-central F interval, labelled η²p whatever univariate measure was chosen, since an η² recovered from a test F is partial by construction; the energy test carries R²ₑ instead, the DISCO partition of its own distances. Roy’s largest root reaches its F through an upper-bound approximation exact only at s = 1, so its p is a lower bound – as SPSS, SAS and base R report it – and the row carries the caveat as a tooltip and again as visible text, because a title attribute survives neither an export nor print.

The univariate follow-ups are fitted on the omnibus’s rows, matched to its kind, and adjusted as their own family. The per-variable fits run on the listwise-complete rows the multivariate fit kept, so their df describe the sample the omnibus protected rather than a larger one. The kind follows the omnibus – Welch’s ANOVAs after Johansen’s, since a pooled follow-up would re-import the equal-covariance assumption it dropped; per-variable energy tests after the energy test, which share its null and assume nothing it did not; mixed and repeated-measures ANOVAs after their multivariate forms – and the heading says which. The K p-values of one multivariate effect are adjusted together by the configured method and never pooled with the global family: the multivariate test is the protected gate, and the note above the table records K and the method so the two families cannot be confused. The internal partial η² stands in only under the η²p header – the producers return NA rather than a partial-η² substitute for ω², ε² or η², so a value is never relabelled. On Mixed MANOVA the per-effect “omnibus was not significant” warning reads that variable’s own follow-up p, because the contrasts are univariate and the multivariate omnibus gates the set, not one variable’s pairs.

The discriminant functions are the eigenvectors of E⁻¹H, solved whitened and reported in the within-group metric. eigen(solve(E) %*% H) is a non-symmetric problem and returns complex values for a spectrum that is real by construction, so E is whitened by its Cholesky factor and the symmetric problem solved instead; each function is scaled to within-group variance 1 and oriented so that its largest standardized coefficient is positive, which keeps signs stable across runs and changes nothing else. The structure coefficients are pooled within-group correlations – SPSS’s Structure Matrix – not the total-sample correlations candisc reports (0.07 apart on a three-group, three-variable check), because a function is defined in the within-group metric and the total-sample version mixes the separation back into its own description; they are shown after the standardized coefficients because they rank the variables as the univariate tests do (Rencher 1988), the blindness the analysis exists to fix. The sequential tests are Bartlett’s χ² of functions m to last, with the multiplier written on the residual df rather than on N so that a MANCOVA’s covariates take their degrees of freedom out; under MANCOVA H and E are the covariate-adjusted Type III matrices, so the centroids are the adjusted means projected onto the functions. A hypothesis df that does not match the group count means the matrices are not this fit’s – a failed statement earlier in the R block leaves the previous run’s in scope – and the block refuses rather than describe another fit.

Each pair’s slope difference and region come from its own moderated refit, at the interval’s confidence level. The fitted model leaves the covariate × factor term out, which is standard ANCOVA and the whole story only where the levels share a slope. Under treatment contrasts a pair is not a linear combination of the full fit’s coefficients, so each pair is releveled to its second member and refitted as dv ~ other covariates + covariate * factor, the interaction term giving the slope-difference test and the Johnson-Neyman region coming off the same fit. A within-subjects pair uses the per-subject difference score, d ~ covariate, whose intercept is the condition difference at the covariate and whose slope is the slope difference, the other covariates centred so the intercept reads at their means; a mixed design runs the condition pairs inside each between cell and the group pairs at each condition – a pair that changes both has neither a difference score nor two independent groups behind it – and tags each block with its side and cell. The region solves θ(m)² = t²·Var(θ(m)) as a quadratic in the covariate (Johnson and Neyman 1936; Bauer and Curran 2005) with t at the confidence level on the fit’s residual df; only boundaries inside the observed covariate range are printed, the share of cases is counted over the actual data rather than the plotted grid, and the plot’s shading comes from the same inequality, so table and plot agree by construction. A plot is drawn where the pair’s own slope test rejects at the assumption test significance level, since the follow-up exists to answer the failed slope-homogeneity check and the omnibus interaction that check tests cannot say which pair – or for every pair on request. On the multivariate cards the regions are per dependent variable, a multivariate region being no standard construct; reporting them beats omitting the follow-up because the omnibus has none.

A rank-deficient response matrix is refused before the fit. The independent MANOVA and MANCOVA check the rank of the dependent-variable matrix on the complete rows as the residuals of lm(dvs ~ grouping + covariates) – so a pair collinear only within the groups is caught, and a variable that reproduces a covariate – and the within designs check their subject × cell response matrix; below full rank the error matrix is singular and car::Anova would fail inside the run, so the check stops it first with the rank and the variable count, and MANCOVA also stops when the complete-case error df, N − groups − covariates, fall below the number of dependent variables. The repeated and mixed designs’ subject bounds are the choosing-a-test note’s.

P-value adjustment #

One pool holds every omnibus p; the post-hoc, follow-up and AUC families are corrected apart from it. The pool is one slot per hypothesis the user could claim significance on – a dependent variable’s omnibus, each pair of an automatic two-sample expansion (which prints no omnibus row and is the primary result), each non-covariate effect row, one representative p per multivariate effect – and one p.adjust call over every slot of the run, batch and pseudo-factorial iterations included, so the family is the study rather than the card. A within-subjects effect contributes its Greenhouse-Geisser or Huynh-Feldt p when sphericity fails, the value its verdict reads, so the adjusted p and the verdict never rest on different numbers; the four multivariate statistics test one hypothesis, so one p enters and the adjusted value is written back to all four rows. Three families stay out. The post-hoc tables and an omnibus categorical test’s opt-in pairwise breakdown are gated by their own omnibus and corrected as their own family under the same method – pooling them would count the omnibus hypothesis twice and inflate the family for every other test in the run, so that switching a breakdown on would tighten the correction on tests it has nothing to do with; a method with its own control (the exact multivariate t, Tukey, Games-Howell, Nemenyi, Dunnett’s tests) takes none. A multivariate test’s univariate follow-ups are protected gatekeeping – the omnibus is the gate, the per-variable p-values one family per multivariate effect – and a one-test family is left raw. DeLong’s AUC comparisons ask a different question from a group difference, so they form their own pool across every variable pair, group pair and batch iteration rather than several small per-call ones, which would under-correct. A covariate is a nuisance control, not a comparison the user makes, so its row takes no share of the error budget – pooling it would tighten the correction on the effect of interest for having controlled for something – and is reported raw. An equivalence run pools the decision p (TOST, non-inferiority, superiority, MET) and runs the two-sided p and, for a one-sided pairwise family, the mirrored p through parallel pools of the same size, so the four-way verdict crosses two decisions controlled at the same level.

The missing-adjustment warning fires only on a family nothing else corrects. None is the shipped default, so a run with an uncorrected family means nobody chose rather than a choice of raw p-values – but a warning that fired on output already adjusted would teach the user to dismiss it. The gate is therefore the pool’s own shape: every batch and pseudo-factorial run; a single run with several dependent variables under a test that fits them separately, with two or more factors, with a two-sample test on a factor of three or more levels, or with post-hoc tests requested by a test that supports them, is not two-sample, and under a method that takes the run-wide adjustment. A self-correcting method covers every contrast table its card draws, and the omnibus rows it leaves over are a family only from two factors up, which the factor count has already caught – so a two-group t-test and a one-way ANOVA under Tukey run silent.

Visualization #

The distribution plots split on the full cell crossing of a factorial or mixed design. A native factorial test consumes every between factor in one model, and a mixed test every factor on both sides, so splitting the box, violin, ECDF and mean and error bar plots on the first factor alone would pool the picture across the others – and, in a mixed design, pool away the within factor the design exists for. The plots therefore split on a synthetic crossed-cell column built the way the combined-factor strategy builds its factor, decoded to its cell labels, with the axis naming the crossed variables; the same column keys the within-subject interval of the mean and error bar plot, each factor flagged within or between, which is why a mixed design’s cells carry it as a dependent design’s conditions do. Where the crossing runs over the cell cap the split falls back to the first factor, and a batch run splits on the iteration’s own factor rather than the first selected.

The forest plot’s diamonds scale by inverse variance, and a ratio measure takes a log axis. The meta-analysis convention: a row’s weight is 1 over its spread squared – the analytic standard error where every row reports one, the interval’s half-width otherwise, measured in log units on a log axis so that [1, 100] and [0.01, 1] rank as equally precise – normalised to the smallest spread and clipped to [0, 1], the diamond running from 3 px up to two-fifths of the row height. The smallest spread is floored at 5% of the median spread, so one near-zero-width row cannot drive every other diamond to a dot. A measure whose null value is 1 – the variance ratio, an odds ratio – is multiplicative, so its axis is logarithmic with the intervals log-spaced rather than forced symmetric; a bound at or below zero, which an estimated ratio interval can produce, breaks a log scale, so the plot then falls back to a linear axis rather than fail. A measure declaring no null value gets neither the reference line nor the stretch to reach it, since an axis dragged out to 0 for means around 120 leaves every interval a dot.

The interaction plot draws adjusted means only when the model’s crossing is the pair drawn. A covariate model prints adjusted cell means on its card and its omnibus tested that adjusted fit, so raw cell means on the plot would contradict both – but only the two-way crossing the estimated marginal means were built on can be read off them, since a higher-order crossing is not its two-way marginal, and a plot that mixed adjusted and raw cells on one axis would read as neither. Every other pair is drawn on raw means, with the Cousineau-Morey interval where one factor is within-subjects and the bars are on. A trace missing an interior level – routine in an unbalanced design – is drawn as separate runs rather than one line joined across the gap, which would read the two non-adjacent levels as neighbours; a cell of one observation is dropped only while bars are drawn, since df = 0 gives a zero-width interval that reads as false precision, and kept as a plain mean otherwise, where dropping it would punch a hole in the pattern.

The category proportion chart’s mark follows the shape of the test, and the chart stacks past 25 categories. The chart and the frequency table on one card show the same counts, and until the marks only the table said which cells drive the result, so the bars carry the table’s own quantity at the table’s own critical z, Bonferroni-adjusted over the family the note names. For the chi-square test that is the cell’s standardized residual – chisq.test’s stdres, the asymptotically N(0, 1) one, not the Pearson residual, whose variance is below 1 and under-flags a cell-wise |z| > 1.96 reading; Fisher’s exact test, Cochran-Armitage and the per-stratum CMH tables return none and stay unmarked. The paired two-condition shapes (McNemar, Stuart-Maxwell) draw two margins of one paired table, so a cell residual is not their hypothesis; the quantity is the category’s standardized marginal change, condition 2 − condition 1 over $\sqrt{{\sum }_{j\ne i}(n_{ij}+n_{ji})}$ – the diagonal of the covariance matrix Stuart-Maxwell inverts, and deliberately not the full multinomial variance, since at k = 2 this makes z² exactly the uncorrected McNemar χ² printed beside the chart – drawn as a connector across the pair with the glyph at its midpoint, where a per-bar mark would read as a residual and any placement off the line would park the glyph on another category’s bar. Cochran’s Q decomposes per condition, $z_j=(C_j-\bar{C})\sqrt{Q/{\sum }_m{(C_m-\bar{C})}^2}$, so that one category’s squared departures sum to Q – the identity recovers the per-subject scale the table does not carry – and a category that never varies across the conditions stays unmarked rather than divide by zero. The family the critical value is adjusted over is the count of quantities, not of glyphs: a binary variable’s second row mirrors the first, so Cochran’s Q marks two bars off one per-condition decision. Past the bar chart’s 25-category ceiling the grouped layout splits each group’s band into sub-pixel bars, so the chart becomes one 100%-stacked bar per group; the tail is not folded into an Other slice because the chart’s job is to show the distribution the contingency test analysed, and a folded tail is one it never saw.

The discriminant plot thins above 5,000 cases and equalizes the pixel scale of its two functions. One circle per case makes the SVG itself the bottleneck on a large frame – the same ceiling the residual Q-Q plots draw at – so the cloud is thinned by a stride within each group, keeping every group’s share of the sample, and the note reports how many of the total are drawn; the centroids are computed on every case. Both functions are scaled to within-group variance 1, so a unit means the same on either axis, and the plot widens the tighter axis to match rather than stretch one function’s separation to fill a card-shaped frame – a gap of 1 is one within-group SD on either. A single function has no second axis, so the groups are banded down the y-axis by a deterministic golden-ratio spread, which puts every point back where it was on a redraw.

The correspondence map places labels by inertia and drops the least where they overlap. Label boxes are estimated from character count rather than measured, since the plot is built before the card is laid out and placement has to be decided in the draw that positions the marks; each label tries eight offsets around its point, takes the first that clears every box already placed, and is dropped when none does. Placement runs heaviest-inertia first, so a crowded map keeps the categories that carry the association and loses the ones near the origin – which are the ones the map says least about – and the note counts the dropped labels against the coordinate tables, which name every category. Each series is drawn in its own principal coordinates, the symmetric scaling that is the default in most software and the most readable, with its caveat printed under the plot rather than left to the reader: a distance within a series reads directly, one between two series does not. The two axes share one pixel scale, since both are inertia units and a card-shaped aspect ratio would stretch one dimension’s spread until no distance was readable.

A mirrored cutoff is reflected onto the ROC curve through its centre. A variable that discriminates in reverse under Compare AUCs keeps its orientation and takes its threshold from the reversed decision rule, so the marker’s sensitivity and specificity belong to that rule; the same threshold sits on the curve drawn at its reflection – false positive rate = specificity, true positive rate = 1 − sensitivity – and the tooltip says that scores at or below the threshold favour the predicted group.

Reproducibility #

A permutation or simulated p-value runs under a seed of its own; the Bootstrap seed governs the intervals alone. The Bootstrap seed is empty by default – fresh draws each run – which is the right default for an interval, where the resampling noise moves a bound in its last digit, and the wrong one for a p-value, which run-to-run drift can carry across α: emmeans integrates the exact multivariate t over the contrasts’ realised correlation by Genz–Bretz Monte-Carlo, and unseeded runs were measured to drift by 1.2 × 10⁻⁴, enough to move a p across a threshold. So every simulated or permutation p – Fisher’s exact test past its exact bounds and the goodness-of-fit test on thin expected counts, the Anderson-Darling, Jonckheere-Terpstra, Jensen-Shannon and energy tests, the symmetry check – and the Monte-Carlo integrations behind Dunnett’s test and the multivariate t run under constants set in the code, offset per dependent variable and per pairwise post-hoc wherever one adjustment pools several p-values, so that p-values corrected together never share a permutation stream; each restores the ambient generator afterwards, so the bootstrap intervals beside them stay under the Bootstrap seed. The one draw that follows the Reproducibility seed is ξ’s balanced min-n subsample, because it is part of the estimator rather than of an inference. A given dataset therefore reproduces its p-values across runs and across the seed settings, and the console’s code names the seed each step ran under.