Effect sizes #

A p-value says whether the data are hard to square with “nothing is going on”. It does not say how much is going on, and that is usually the question that was asked. An effect size is the answer: one number for the magnitude of a difference or a relationship, on a scale that can be compared across studies. Every results table in DataSuite that carries a p can carry one beside it, under the header Effect, Effect size or the measure’s own symbol. This page explains the families of measures, what their numbers mean, and how to read the “small”, “medium” and “large” labels the app attaches. The numbers come from the café data, prepared the way that page describes.

Why a p-value is not enough #

Two results from the café. Station members spent more per visit than Riverside members before the menu change: the gap is 1.61, and Welch’s t-test gives p = 0.002. Members also made more visits after the change than before: 0.64 more visits per member over eight weeks, and the paired t-test gives p = 0.051. Read by p alone, the first is a finding and the second is not. Read by size they are closer than they look: the spend gap is a little under half a standard deviation (d = −0.45), the change in visits a seventh of one (dz = 0.14) – and the second comes as close to significance as it does only because 200 pairs of measurements make even a slight change detectable.

A p-value depends on the size of the effect and the size of the sample. Double the sample and a fixed effect’s p shrinks; a trivial effect reaches p < 0.001 given enough cases, and a large one misses 0.05 in a study of twelve. The effect size takes the sample size out: it describes what was found, not how sure you are of it. A report needs both, with a confidence interval on the effect size to carry the uncertainty – the interval says how large the effect could plausibly be, which p never does.

Effect is the column that holds the effect size in some of the app’s tables, under whichever measure was chosen; the table’s caption, or the effect size option that produced it, names which.

Effect size is a standardised measure of how large a difference or a relationship is, independent of the sample size: a difference in standard-deviation units, a correlation, a share of variance explained, an odds ratio. The families below are the ones the app computes. Which one a table shows depends on the test, and the module’s manual lists the pairing – comparison analysis has the longest list.

The difference itself #

Difference is the simplest effect size there is: the gap between two means (or medians, or proportions) in the units the variable was measured in. Riverside minus Station on spend_before is −1.61 per visit; spend per visit rose by 1.09 after the menu change, 12.6% of the before average. The app reports it with a confidence interval beside every mean-comparison test, in the order the groups are listed – so a negative difference means the second group is higher. When the units mean something to the reader – money, minutes, points on a scale they know – this is the effect size to lead with; the standardised measures below exist for comparing across studies that measured in different units, and for the cases where the units mean nothing on their own.

Standardised mean differences – the d family #

Cohen’s d is the difference between two means divided by their pooled standard deviation: the difference expressed as “how many SDs apart”. For spend_before by branch the gap of 1.61 over a pooled SD of 3.54 gives d = −0.45, the sign following the group order (Riverside minus Station). It is the measure the independent-samples t-test pairs with, and the one most benchmarks were written for. What it means concretely: the average Station member sits at the 67th percentile of Riverside’s spending, and a Station member picked at random outspends a Riverside member picked at random 63 times in 100. Cohen’s benchmarks – 0.2 small, 0.5 medium, 0.8 large – are the bands the app’s Interpretation column applies to the whole family.

Hedges’ g is Cohen’s d with a small-sample correction: d overestimates the population effect slightly, by more the smaller the sample, and g shrinks it by a factor set by the degrees of freedom. On the café branches g = −0.453 against d = −0.454 – with 200 cases the correction lives in the third decimal and the two are interchangeable; below thirty or so cases, report g.

Welch’s d divides by the root mean square of the two groups’ SDs rather than by their pooled SD, and is the measure Welch’s t-test offers – pooling the spreads would contradict the reason for choosing Welch’s, which is that they differ. On the café branches (SDs 2.97 and 4.05) the two denominators are 3.54 and 3.55 and Welch’s d = −0.454, the same as Cohen’s to two decimals; they diverge when the groups are unequal in size as well as in spread.

Welch’s g is Welch’s d with the bias correction Hedges’ g applies, taken at Welch’s degrees of freedom: −0.452 on the café branches.

Glass’ Δ divides the difference by one group’s SD alone, for the case where the groups’ spreads are so different that neither an average nor a pool describes both – typically a control group whose spread the treatment did not touch. The app reports both versions, since it has no control-group selector: Δ₁ uses group 1’s SD and Δ₂ group 2’s. On the café branches they are −0.54 (by Riverside’s 2.97) and −0.40 (by Station’s 4.05) – the same 1.61 read against two yardsticks, and a reminder that a standardised difference is only as meaningful as its denominator.

Cohen’s dz is the paired-design d: the mean of the within-pair changes divided by the standard deviation of those changes. Spend per visit rose by 1.09 on average, with the changes spread with an SD of 2.86, so dz = 0.38; visits rose by 0.64 with an SD of 4.61, dz = 0.14. It is the measure the paired t-test pairs with, and it is not comparable with a d from an independent design: its denominator is the spread of the changes, which is smaller than the spread of the values when before and after are correlated (r = 0.69 here), so the same mean change reads larger as dz.

Hedges’ gz is Cohen’s dz with the small-sample correction: 0.379 against 0.381 on the café spend.

Cohen’s d_av is the paired-design alternative that divides by the average of the two conditions’ SDs instead of the SD of the changes, so that it lives on the same scale as an independent-groups d: 0.30 for the café spend, against dz = 0.38. Report d_av when the reader will compare the effect with between-groups studies, dz when the question is how consistently individuals changed.

Hedges’ g_av is Cohen’s d_av with the small-sample correction, the factor Hedges’ g applies, taken at n − 1 degrees of freedom for n pairs: 0.298 against d_av = 0.299 on the café spend, the correction living in the third decimal at 199 pairs as it does for g against d. It is the paired measure to pool in a meta-analysis, where the bias it removes accumulates across small studies.

Correlations and variance explained – the r family #

r is the correlation coefficient: how closely two variables move together, from −1 (one rises exactly as the other falls) through 0 (no linear relationship) to +1. Overall satisfaction and the taste item correlate at r = 0.55 on the café survey; satisfaction and waiting time at r = −0.30 – the longer the wait, the lower the score. As an effect size r reads on Cohen’s benchmarks of 0.1, 0.3 and 0.5, and the app’s correlation strength bands add “very strong” above 0.7 and “very weak” below 0.1. The same header appears in a comparison table when the chosen measure is a correlation-type effect – point-biserial, rank-biserial, Wilcoxon r or, as r₊, the sign test’s r – read on the same scale.

Point-biserial r is the correlation between the dependent variable and group membership coded 0 and 1: the two-group difference restated as a correlation. For spend_before by branch it is 0.22 – small on r’s benchmarks, as d = −0.45 is small on d’s. The two describe one effect on two scales, and $d=2r/\sqrt{1-r^2}$ converts one into the other when the groups are equal in size.

r² is r squared: the share of one variable’s variance that the other accounts for. Overall satisfaction and taste share r² = 0.30 of their variance, satisfaction and waiting time 0.09. It is the r-family measure that regression reports as R², and the one to use when the question is “how much of the variation does this explain” rather than “how strong is the association”.

η² (eta squared) is the share of the dependent variable’s variance that the groups account for – r² for a comparison of any number of groups, and for two groups exactly the point-biserial r². Branch accounts for η² = 0.050 of the variance in spend_before: 5%, which the variance family’s benchmarks (0.01 small, 0.06 medium, 0.14 large) call small. It is the measure the one-way and factorial ANOVAs offer, and it overstates the population share by an amount that shrinks as the sample grows.

ω² (omega squared) is η² corrected for that overstatement: an estimate of the share of variance explained in the population rather than in the sample. On the café branches ω² = 0.045 against η² = 0.050; with ten cases per group the gap would be several times wider. Prefer it in a report; the benchmarks are the same.

ε² (epsilon squared) is a second correction of the same kind that lands between η² and ω², offered beside them for a one-way ANOVA. The Kruskal-Wallis test pairs with an ε² of its own, computed on the ranks – $H/(n-1)$, the share of the rank variance the groups explain, with no correction – and with η²_H as its corrected form; the symbol serves both.

Eta-squared (H) – η²_H – is the Kruskal-Wallis test’s corrected rank measure, $(H-k+1)/(n-k)$: H less the k − 1 it is expected to reach when the groups do not differ, over the n − k left to explain – the rank analogue of ω²’s bias correction, clamped at 0. Waiting time across the three age bands, H(2) = 1.92, p = .383 on medians of 4.45, 5.00 and 4.10 minutes: ε² = .011 and η²_H = 0, the rank family on a null – a test below its own expectation prints a 0 on the corrected scale where the uncorrected one still shows a share. Both read on the variance family’s benchmarks, with a bootstrap interval when effect-size intervals are on.

Partial η² is η² with the other effects of a factorial design taken out of the denominator: the share of the variance not already explained by the other factors that this factor explains. For the same effect it is larger than plain η² whenever there is more than one factor, so the two must never be compared across designs. It is what the repeated-measures and mixed ANOVAs report, and the benchmarks are the variance family’s, read with that caveat.

Variance explained in factorial and repeated-measures designs #

With more than one factor, or the same members measured more than once, “the share of the variance this effect explains” has more than one possible denominator, and the measures below differ in what they leave in it. The running instance is the before-and-after change in spend as a one-factor repeated-measures ANOVA on 199 members: F(1, 198) = 28.84, p < .001, the paired t-test’s t of 5.37 squared.

Generalized eta-squared – η²G – is the effect’s sum of squares over the effect plus every source of measured variance in the design, the differences between the members included, so that a within-subjects effect is read against the same total a between-groups design would use and the two can be compared. For the change in spend η²G = .022, against partial η² = .127 for the same effect, whose denominator keeps only the spread of the changes and drops the far larger spread between members – the number a repeated-measures design flatters. Quote the generalized one beside a between-groups study and the partial one within the design (Olejnik & Algina 2003; Bakeman 2005). It prints without an interval, since the subject variance in its denominator is not bounded by the effect’s F, and reads on the variance family’s benchmarks.

Partial omega-squared – ω²p – is partial η² with the bias correction that takes ω² from η²: an estimate of the partial share in the population rather than in the sample. ω²p = .122 for the change in spend against η²p = .127, and the two share the interval, .05 to .21, that the non-central F bounds for the population quantity both estimate. It is the least biased partial measure the repeated-measures and factorial ANOVAs offer, and partial η²’s caveat carries over: a partial share is never compared across designs with different numbers of factors.

Multivariate partial η² is the effect size of a MANOVA’s omnibus test: the share of variance the grouping accounts for across the dependent variables taken together, on the combination of them that separates the groups best, recovered from Pillai’s approximate F and printed on the Pillai row under η²p – the partial form always, whatever measure the univariate follow-ups were set to. overall, wait and spend_before together by branch: Pillai’s trace .098, F(3, 174) = 6.28, p < .001, and η²p = .098 [.02, .18] – with two groups the trace and the effect size are the same number, since there is only one discriminant function to separate them on. It reads on the variance family’s benchmarks, and the follow-up tests carry their own per-variable measure.

Robust measures – the trimmed-means family #

Yuen’s test and the trimmed-means ANOVA compare trimmed means, and a d or an η² computed on the untrimmed values would describe a contrast the test never made; the Welch and Brown-Forsythe ANOVAs drop the equal-variance premise that η² and ω² partition under. Two measures are built for them.

Explanatory measure of effect size – ξ – is the robust counterpart of √η²: the spread of the groups’ trimmed means as a share of a robust estimate of the total spread, the winsorized variance of the pooled sample rescaled to the ordinary variance’s scale, on a 0 to 1 range with benchmarks of 0.10, 0.30 and 0.50 (Mair & Wilcox 2020). spend_before by branch under Yuen’s test gives ξ = .31 – medium on those benchmarks, only just, where Cohen’s d = −0.45 is small – and across the three age bands the trimmed-means ANOVA gives ξ = .38. It is unsigned, so it says how far apart the groups are and not which is higher, and it is defined for groups of equal size: with unequal ones the app takes the median of the measure over equal-sized draws from the groups, and the bootstrap interval it prints when effect-size intervals are on – roughly .10 to .50 for the branches – is marked as a conservative one. The Welch and Brown-Forsythe ANOVAs, which compare untrimmed means, take it at a trim of zero.

Algina-Keselman-Penfield robust d – δ_t – is Cohen’s d built from the same robust quantities: the difference between the two trimmed means over the pooled winsorized SD, times the constant (0.642 at the default 20% trim) that puts a winsorized SD back on the ordinary SD’s scale, so that Cohen’s 0.2, 0.5 and 0.8 apply. It is the signed measure Yuen’s test offers beside the unsigned ξ, and it describes the contrast the test made: δ_t = −0.43 [−0.73, −0.14] on spend_before by branch, beside Cohen’s d = −0.45 on the untrimmed means – the two part company when the tails are heavy, which is when the trimmed test was chosen. Its interval is built on Yuen’s own standard error, so it excludes zero exactly when the test rejects, and it is defined for two groups only.

Rank-based measures #

Waiting time is skewed – most waits are short and a few are long – so the comparison manual sends it to the Mann-Whitney test rather than a t-test. The branches’ median waits are 4.8 and 5.0 minutes, U = 3889, p = 0.289: no evidence of a difference. The rank-based effect sizes say how far apart the branches are anyway, without assuming anything about the shape of the waits.

Cliff’s δ is the probability that a value from group 1 exceeds a value from group 2, minus the probability of the reverse – every Riverside wait set against every Station wait, 95 × 90 pairs. It runs from −1 (every value in group 1 below every value in group 2) through 0 to +1, and on the café waits it is −0.09: Station’s waits are the longer ones slightly more often than not. Its benchmarks are 0.147, 0.33 and 0.474 (Romano et al. 2006), so −0.09 is negligible, and the interval the app puts beside it, −0.25 to 0.08, straddles zero.

Rank-biserial r is Cliff’s δ under another name – for two independent groups the two are the same number, $1-2U/(n_1{n}_2)$, and the app reports the same value and interval for either. The name signals which scale to read it on: r’s benchmarks (0.1, 0.3, 0.5) rather than δ’s. In a paired design its counterpart is the matched-pairs rank-biserial r, built from the ranks of the within-pair changes.

Common language ES is the probability, read straight off the pairs, that a value from group 1 is larger than a value from group 2: 0.455 on the café waits – a Riverside member’s wait beats a Station member’s in 45.5 pairs out of 100, a tie counting as half. It is Cliff’s δ rescaled to a probability, $(\delta +1)/2$, and for a d-family effect its normal-theory twin is what the Cohen’s d entry quoted: 63 in 100. It reads on McGraw & Wong’s benchmarks of 0.56, 0.64 and 0.71 – 0.5 is no effect.

Matched-pairs rank-biserial r is the paired design’s rank-biserial r, the Wilcoxon signed-rank test’s companion: rank the sizes of the within-pair changes, and take the share of the rank total held by the members who went up minus the share held by those who went down, $r_{rb}=(W^{+}-W^{-})/(W^{+}+W^{-})$, the pairs that did not change left out. It runs from −1 (every member went down) through 0 to +1 (every member went up). On the change in spend, W = 13,850 over the 197 members whose spend changed at all, and r_rb = .42 – the members who spent more after the menu change did so by larger margins, rank for rank, than those who spent less – medium on r’s benchmarks, with a bootstrap interval. The one-sample Wilcoxon test reports the same measure on the deviations from the reference value.

Wilcoxon r is the signed-rank test’s other companion, $r=Z/\sqrt{N}$: the test’s standardised statistic over the square root of the number of pairs it ranked, on the same −1 to +1 scale and the same benchmarks. On the change in spend it is .36, against a matched-pairs rank-biserial of .42. The two are not interchangeable – r_rb reads the balance of the ranks directly, while Wilcoxon r restates the test statistic’s distance from its null and so carries the tie and continuity corrections the p-value does – so a report says which one it quotes. The one-sample test offers it too, on the deviations from the reference value.

Sign-test r (2p₊−1) is the sign test’s companion, paired or one-sample: the share of pairs that went up, p₊, restated as 2p₊ − 1 so that it runs from −1 to +1 – 0 is half each way, +1 every pair up – with a Wilson interval on p₊ carried onto the same scale. Of the 197 members whose spend changed, 125 spent more: p₊ = .63 and r₊ = .27 [.13, .40]. The waits against the five-minute reference, 87 of 183 above it, give r₊ = −.05. Its benchmarks are 0.10, 0.30 and 0.50 – Cohen’s g’s doubled, since it is the same departure of a proportion from a half on a scale twice as wide.

Kendall’s W is the Friedman test’s companion: the agreement among the subjects about the order of the conditions, from 0 (each member ranks the conditions differently) to 1 (every member ranks them the same way), $W={\chi }^2/(N(k-1))$ from the test’s own statistic. Take the six satisfaction items as six conditions, slow reverse-scored, each member ranking them by the score given: Friedman’s χ²(5) = 21.16, p < .001, and W = .021 – the items do differ, friendliness rated highest at 3.79 and value lowest at 3.34, but the members agree about the order hardly at all, and 200 of them make a significant test out of that. The app labels it as agreement on Schmidt’s (1997) bands for the coefficient – very weak below 0.3, then weak, moderate, strong and unusually strong from 0.3, 0.5, 0.7 and 0.9 – rather than as an effect on r’s benchmarks, since W is a coefficient of concordance and not a correlation; the agreement page reads the same bands, so the coefficient meets one verdict wherever it is reported. The interval is a bootstrap over the subjects.

Kendall’s tau-b is the Jonckheere-Terpstra trend test’s companion: Kendall’s τ between the ordered group and the outcome, in the τ-b form that discounts the tied pairs an ordered factor is full of, so that the trend the test found is restated as a correlation from −1 to +1 on r’s benchmarks. spend_before across the age bands coded 1, 2 and 3: J = 7,306.5, p = .002 and τ-b = .20 – a small rising trend, the means climbing from 6.85 through 9.07 to 9.48 – with a bootstrap interval drawn within the groups.

Mean within-block ρ – ρ̄ – is Page’s trend test’s companion: for each subject, Spearman’s ρ between the hypothesised order of the conditions and that subject’s actual ranking of them, averaged over the subjects. It is +1 when every subject’s ranking follows the order exactly, 0 when the rankings say nothing about it and −1 when every one runs against it, read on r’s benchmarks with a bootstrap interval over the subjects; a subject whose scores tie across every condition contributes a 0. The café has no ordered conditions to try it on – a dose taken at three levels by every patient, the response ranked within each, is its shape, and a ρ̄ of .8 would say that nearly every patient’s response climbed with the dose.

Measures for categorical outcomes #

The survey asked whether the member would recommend the café. 73 of Riverside’s 101 members said yes (72.3%) and 62 of Station’s 99 (62.6%); the chi-square test gives χ²(1) = 2.12, p = 0.145 with the continuity correction off (the app applies it by default and prints 1.71, p = 0.192) – no evidence either way that the branches differ. The plain difference is 9.7 percentage points; the measures below put it on scales that carry across tables.

Cramér’s V is the chi-square statistic rescaled to the sample size and the table’s shape,

so that it runs from 0 (no association) to 1 whatever the table’s dimensions: √(2.12 ⁄ 200) = 0.10 for recommendation by branch. On a 2×2 table it equals the φ coefficient, the correlation between the two yes/no variables. The benchmarks are Cohen’s w’s – 0.1, 0.3, 0.5 – for a 2×2 table, and the app divides them by √df for larger tables, since the same V means more on a 4×4; 0.10 is small, and only just.

Odds ratio is the odds of the outcome in one group divided by the odds in the other. At Riverside 73 yes to 28 no is odds of 2.61 to 1; at Station 62 to 37 is 1.68 to 1; the ratio is 1.56 – Riverside members’ odds of recommending are 1.56 times Station’s. The app prints 0.64 for this table, the reciprocal, because it takes the cross-product of the table in the order its rows and columns are listed (no before yes, Riverside before Station) – 0.64 is Riverside’s odds of a no relative to Station’s, and 1 ⁄ 0.64 = 1.56 its odds of a yes; the two are the same finding, and the interval, 0.35 to 1.17, contains 1, which is no association. Read the magnitude on the larger of OR and 1 ⁄ OR: 1.68 small, 3.47 medium, 6.71 large (Chen et al. 2010). An odds ratio is not the ratio of the two proportions – that is 72.3 ⁄ 62.6 = 1.15, the relative risk – and it drifts away from it as the outcome gets more common, so “1.56 times the odds” must not be read as “1.56 times as likely”. On a Cochran-Mantel-Haenszel row the same column holds the common odds ratio, one ratio pooled over the strata: .65 [.35, 1.18] for recommendation by branch within the age bands.

Cohen’s w is the chi-square statistic rescaled to the sample size alone, $\sqrt{{\chi }^2/n}$, for a goodness-of-fit test of one categorical variable against expected proportions. Testing whether the 135 yes and 65 no are an even split gives χ²(1) = 24.5, so w = √(24.5 ⁄ 200) = 0.35 – medium on the benchmarks 0.1, 0.3 and 0.5, which are the same as Cramér’s V’s because on a 2×2 table the two coincide.

Cohen’s h is the difference between two proportions after each is transformed to an angle ($2\arcsin \sqrt{p}$), a transformation that makes a given gap count the same near 0.5 as near the extremes, where a raw percentage-point difference is worth more. The one-sample proportion test reports it for the observed share against the hypothesised one: 67.5% recommending against a hypothesised 50% is h = 0.36; between the branches, 72.3% against 62.6%, it would be 0.21. It reads on Cohen’s d benchmarks – 0.2, 0.5, 0.8.

Measures for paired categorical outcomes #

The paired categorical tests – McNemar’s, Stuart-Maxwell, Cochran’s Q – ask whether the same subjects’ answers moved between occasions, and their effect sizes measure that movement, not an association between two variables. The café asked its members once, so the instances are from a clinic: 200 patients asked whether a symptom is present at two visits, of whom 30 lost it and 10 gained it, and 150 patients asked at three visits.

φ (marginal) is $\sqrt{{\chi }_{\text{McNemar}}^2/N}$: McNemar’s chi-square on the discordant pairs, rescaled to the sample size, on a 0 to 1 scale – (30 − 10)² ⁄ 40 = 10 for the clinic, and √(10 ⁄ 200) = .22. It is not the φ coefficient of a 2×2 table, which measures how strongly the two occasions’ answers agree; it measures how far the margins moved, and the two can be far apart on one table – patients who all kept their answer give an agreement of 1 and a marginal φ of 0. It carries no small-medium-large label, since no benchmark was ever calibrated for it; Cohen’s g below is the calibrated measure of the same change.

Cohen’s g is the departure of the switchers’ split from a half: with b patients moving one way and c the other, $g=|b-c|/(2(b+c))$, from 0 (an even split, the margins unchanged) to 0.5 (every switcher went the same way). The clinic’s 30 against 10 gives 20 ⁄ 80 = .25, large on the benchmarks Cohen set for it, 0.05, 0.15 and 0.25. On an answer with more than two categories – the Stuart-Maxwell test’s – one g is reported per pair of categories with any switchers between them, since a single aggregate would not be the measure those benchmarks were calibrated on. The interval is a bootstrap over the pairs; when every switcher went the same way the estimate sits at 0.5 and no interval can be drawn.

Cohen’s w (Q) is Cohen’s w for Cochran’s Q, $\sqrt{Q/N}$: the test’s statistic over the number of subjects – all of them, the patients who gave the same answer at every visit included, as a chi-square effect size conventionally counts the sample the test ran on. Q = 12 over the 150 patients gives w = .28, small on w’s benchmarks of 0.1, 0.3 and 0.5; it is unbounded above, and its interval inverts the non-central chi-square.

Cochran’s average φ² – φ̄² – is the same Q over the informative subjects alone and per degree of freedom, $Q/(N_{\text{inf}}(k-1))$, where N_inf counts only the patients whose answer varied across the visits – the count the informative subjects check screens Q on. A patient who answered the same way every time adds nothing to Q and nothing to the denominator, so the measure is the share of the most heterogeneity the changers could show, bounded at 1 and read on the variance family’s benchmarks, 0.01, 0.06 and 0.14, with a bootstrap interval over the subjects. If 60 of the 150 patients ever changed their answer, φ̄² = 12 ⁄ (60 × 2) = .10, medium – where w counted all 150 and read small. The two denominators are the whole difference: w asks how large the change is across the sample, φ̄² how consistent the changers were.

Measures for whole distributions, and for spread #

The whole-distribution tests react to a difference of any kind, and their effect sizes are shares on a 0 to 1 scale with no direction and no small-medium-large label. The mutual information test’s is Theil’s U, defined on the association page: in a comparison it is the mutual information between the group and the outcome over the entropy of the group – the share of the uncertainty about which group a member belongs to that knowing the outcome removes. U = .017 for spend_before by branch, where Cohen’s d reads −0.45, and .087 for the waits, where the location tests find nothing: a shift is spread thinly over every value, a difference in shape is not.

Normalized Jensen-Shannon divergence – JSD_n – is the Jensen-Shannon divergence between the groups’ distributions over its ceiling, log₂ k bits for k groups, so that it runs from 0 (one distribution) to 1 (no overlap at all) whatever the group count, where the raw divergence in bits means one thing for two groups and another for five. The waits by branch give .067. It is a point estimate only, with no interval and no benchmark label.

R²ₑ is the energy test’s effect size: the share of the total dispersion of the standardized dependent variables – every distance between two members – that lies between the groups rather than within them, the decomposition of the same distances the test permutes. The E statistic itself grows with the sample and says nothing across datasets; R²ₑ is bounded between 0 and 1 and does. wait and spend_before together by branch: E = 8.22, p = .001, R²ₑ = .026. No interval and no label.

Variance ratio – VR – is the F-test for equality of variances’ effect size: the first group’s variance over the second’s, with the test’s own confidence interval, and 1 rather than 0 as “no effect”. It is the assumptions page’s variance ratio read as a finding rather than as a check – spend_before by branch, .54 [.36, .80]: Station’s variance is 1.86 times Riverside’s, and the interval excludes 1 – and it carries no small-medium-large label, since no benchmark was ever set for a ratio of spreads; the rule of thumb that a ratio up to four leaves a pooled test intact is a rule about the test, not about the size of the difference. The interval is two-sided even when the test was run one-sided.

The symbols in a results table #

The effect-size column is headed by the measure’s symbol rather than its name, with its interval and standard error under d CI and d SE beside it. The symbols the app uses:

• d – a standardised mean difference: Cohen’s d, Welch’s d or the one-sample d, depending on the test; the effect-size menu says which
• g – the bias-corrected d, Hedges’ g or Welch’s g; on a McNemar or Stuart-Maxwell row it is instead Cohen’s g, the measure of marginal change
• dz – Cohen’s dz, the paired-design d
• gz – Hedges’ gz, its bias-corrected form
• d_av – Cohen’s d_av, the paired-design d on the between-groups scale
• g_av – Hedges’ g_av, the bias-corrected d_av
• Δ₂ – the second of the two Glass’ Δ values, standardised by group 2’s SD; the first sits in the main column
• δ_t – the Algina-Keselman-Penfield robust d, on a trimmed-means row
• ξ – the explanatory measure of effect size of the robust and heteroscedastic tests; the correlation module’s ξ is Chatterjee’s, another measure under the same letter
• r_pb – point-biserial r
• r_rb – rank-biserial r, for two groups the same number as Cliff’s δ; on a signed-rank row the matched-pairs rank-biserial r
• r – Wilcoxon r on a signed-rank row; as r₊, the sign test’s r
• δ – Cliff’s δ on a rank-test row; as a capital Δ on a t-test row, Glass’ Δ
• CL – the common language effect size
• W – Kendall’s W on a Friedman row; as a lower-case w, Cohen’s w on a goodness-of-fit or trend row and Cohen’s w (Q) on a Cochran’s Q row
• τ_b – Kendall’s tau-b, the Jonckheere-Terpstra test’s trend measure
• ρ̄ – the mean within-block ρ, Page’s trend test’s
• η²p – partial η²; on a MANOVA’s Pillai row the multivariate partial η²
• η²G – generalized η², the repeated-measures ANOVA’s variant that compares across designs
• ω²p – partial ω²
• η²_H – the Kruskal-Wallis η²_H, beside ε²
• φ̄² – Cochran’s average φ²
• V – Cramér’s V
• OR – the odds ratio, or on a Cochran-Mantel-Haenszel row the common odds ratio
• h – Cohen’s h
• φ – the marginal φ of a McNemar row, not the association φ coefficient
• U – Theil’s U on a mutual-information row; the Mann-Whitney U is a test statistic and sits in the statistic column
• JSD_n – the normalized Jensen-Shannon divergence
• R²ₑ – the energy test’s R²ₑ
• VR – the variance ratio of the F-test

Reading small, medium and large #

The Interpretation column labels each effect size small, medium or large (with “negligible” below the small threshold) against the benchmarks quoted above. They are Cohen’s, published in 1988 with an explicit apology: they were set so that a “medium” effect would be visible to the naked eye of a careful observer, and he called on every field to replace them with its own. Two cautions follow. First, the label is a reading of the number against a fixed bar, not of the finding against its context: a d of 0.2 in survival after a cheap intervention is enormous, and a d of 0.8 on a test of the thing the training taught is unremarkable. Second, the label says nothing about certainty. A small effect measured in a large sample may be established beyond doubt; a large one in a small sample may be noise, and the confidence interval beside it is what tells the two apart.

The families do not share a scale. A d of 0.45, an r of 0.22, an η² of 0.05 and a common-language effect of 0.63 are, on the café data, one effect described four ways, and each is small on its own benchmarks. Compare effect sizes only within a family, convert between them when a comparison across studies needs it, and say which measure a number is every time it is quoted.

Some measures carry no label at all, and the Interpretation column names no magnitude beside them: the marginal φ, the variance ratio, Theil’s U, the normalized Jensen-Shannon divergence and R²ₑ are shares or ratios for which no benchmark was ever calibrated, and the number is reported as it stands; the MANOVA’s multivariate partial η² is read on the variance family’s benchmarks by the reader, since its table prints the value and its interval alone. Kendall’s W is labelled as agreement – very weak to unusually strong, on Schmidt’s bands – rather than as an effect.

Reporting an effect size #

Report the measure by name, its value, and its confidence interval, next to the test statistic and the p – and, for a mean difference, the raw difference in the original units as well, since it is the number a reader can picture. For the café’s branch comparison: “Station members spent more per visit than Riverside members, 9.52 versus 7.91, a difference of 1.61 (95% CI 0.62 to 2.60), Welch’s d = 0.45 (95% CI 0.17 to 0.74), t(179.5) = 3.20, p = 0.002.” The sign was dropped and the direction stated in words, which is the readable convention; the app’s tables keep the sign, in the order the groups were listed. Each module’s manual has a reporting checklist that names the measure its tests pair with.

Where this appears in DataSuite #

• Comparison analysis – Include effect sizes adds the measure chosen from the menu, with optional confidence intervals and standard errors; every test has its own list, from the d family for t-tests through η² and its relatives for ANOVA to Cramér’s V and the odds ratio for tables.
• Correlation analysis – the coefficient is the effect size, and the strength label follows the correlation bands.
• Regression analysis – R² and adjusted R² for the model, standardised coefficients (β) for the predictors, odds ratios for logistic models.
• Meta-analysis – pools effect sizes across studies: standardised mean differences, correlations, odds and risk ratios.
• Analysis planner – takes the effect size you expect and returns the sample size a test needs to detect it.
• Settings – the strength bands for correlations and their relatives; the d, variance and odds-ratio benchmarks are fixed.