Agreement and reproducibility #

The café’s till recorded what each loyalty member spent per visit in the eight weeks before the menu change and the eight weeks after – two measurements of the same 200 members. The survey asked each of them whether they would recommend the café, and the owner has a rule of thumb that answers the same question from the 0-to-10 overall rating: anyone at 8 or above would. Two measurements of one thing, two classifications of one thing, six ratings of one thing – whether they give the same answer is the question of agreement, and this page explains how it is asked and answered: what agreement is and why a correlation does not measure it, the intraclass correlation and its forms, the kappa family and Krippendorff’s alpha for categories, Kendall’s W for rankings, the error in one repeated score and the change it can detect, the Bland–Altman plot with its limits of agreement, and the signal-detection indices that separate how well a rater tells cases apart from how readily it says yes. The numbers come from the café data, prepared as that page describes, stacked into long format with the table converter – one row per member per condition – and run through the Reproducibility & agreement module. The café is not an agreement study – nothing in it was measured twice by two methods – so where a pair of columns stands in for one, the page says what the stand-in leaves out.

What agreement is #

Agreement is whether two or more measurements of the same thing give the same answer – the same score from two raters, the same reading on two occasions, the same category from two classifiers. It is the reliability question asked with a second measurement in hand, where internal consistency asks it of one questionnaire’s items on one occasion: a scale can hang together perfectly and still be scored differently by two raters, which is why the two are separate modules. In the app’s roles a subject is the thing measured, a condition is who, when or how measured it – a rater, a time point, a method – and every other selected variable is a score the conditions are compared on, one row per subject per condition (wide and long). For the café the subjects are always the members; the conditions are spend_before and spend_after stacked into one column, or the survey’s recommend beside the owner’s overall ≥ 8 rule, or the six satisfaction items read as six raters of one member’s satisfaction. A clinician: patient → subject, two scanners → conditions, the measured volume → score; a teacher: essay → subject, two markers → conditions, the mark → score; a coder of interview transcripts: passage → subject, two coders → conditions, the category → score.

Test–retest reliability is agreement across time: the same subjects measured again under the same conditions, so that any difference is measurement error – the till’s rounding, a member’s ordinary week-to-week variation – and not a change in the thing measured. The spend pair is the café’s nearest instance, and it is not quite one: the menu changed between the two windows, so a member’s after figure carries a real change as well as noise, and every coefficient below that treats the pair as a retest reads that change as disagreement. The page uses the pair anyway, because the arithmetic is the same and the caveat is instructive – a retest interval in which something happened is the commonest flaw in a test–retest study.

Inter-rater reliability is agreement across judges: the same subjects rated by two or more raters at one time, so that any difference is in the raters – how they read the rubric, where they put the boundary between a 3 and a 4. The café has no raters, so two stand-ins serve: recommend beside the owner’s rule are two classifiers of one member, and the six items are six ratings of one member’s satisfaction, each a rater with a scale of its own. When a study has several raters but not the same ones for every subject – whichever nurse was on shift – the design is still inter-rater, and the one-way model below is written for it.

Consistency is agreement about order: two conditions agree in the consistency sense when they rank the subjects the same way, whatever scores they give. A rater who gives every essay two marks more than a colleague is perfectly consistent with them – the best essay is the best for both – and a till that reads one unit high on every visit is perfectly consistent with one that does not. Spend after the menu change is 1.09 units higher than before on average, and the two windows agree in the consistency sense to 0.70 on the two-way mixed ICC; a correlation is a consistency measure too.

Absolute agreement is agreement about value: two conditions agree in the absolute sense when they give the same score to the same subject, so that a constant offset between them counts as disagreement. It is the sense that matters when the conditions are to be used interchangeably – a reading from either scanner going into the same chart, either marker’s mark standing as the grade. The café’s before-and-after spend agrees less well in this sense, 0.67 on the two-way random ICC, because the 1.09 shift is charged against it; the gap between 0.70 and 0.67 is the shift’s size relative to the differences between members, which is small on a scale where the members’ spends have a standard deviation of about 3.6. Whether a study wants consistency or absolute agreement is a decision about how the scores will be used, made before the coefficient is chosen.

Correlation is not agreement. A correlation says whether two measurements move together; agreement asks whether they give the same value, and the first can be perfect while the second fails. Add 5 to every member’s after-spend, as a till that reads five units high would: the correlation between the two windows stays exactly 0.69 and the consistency ICC exactly 0.70, while the absolute-agreement ICC falls from 0.67 to 0.29 and the one-way form to 0.01 – the members are ranked the same, and every one of them is five units apart. Nothing in a correlation coefficient can see the offset, since it is fitted with a free slope and a free intercept; the ICC, kappa and Bland–Altman all can. The module reports Pearson’s r and Spearman’s ρ between two conditions – 0.69 and 0.68 for the spend pair, read on correlation bands rather than agreement bands – as a check on consistency alone; a study whose question is whether two methods can replace one another reads the ICC and the limits of agreement instead.

The intraclass correlation #

Intraclass correlation is the share of the variance in the scores that is differences between subjects, rather than differences between conditions or noise – the reliability idea, true-score variance over total variance, computed from repeated measurements instead of from a scale’s items. Split the variance of the 399 spend figures three ways: 9.41 of it is members differing from one another, 0.55 is the two windows differing – the menu effect – and 4.10 is what is left, a member’s own wobble between the two windows, for a total of 14.06. The ICC is 9.41 ⁄ 14.06 = 0.67: two thirds of the variance in a spend figure is which member it belongs to, and a third is occasion and noise. The app reads it against Koo and Li’s bands – below 0.50 poor, 0.50 to 0.75 moderate, 0.75 to 0.90 good, above 0.90 excellent – so 0.67 is moderate, with an interval of [0.55, 0.76] that reaches into both neighbouring bands. Which variances go into the denominator is what the models and forms below decide, and the literature names the choice ICC(model, form): ICC(2,1) is the two-way random model for a single measurement, and is the one above. An ICC is a reliability in the attenuation formula’s sense too – a test–retest ICC can stand where the planner asks for a measure’s reliability.

One-way random effects model treats each subject’s conditions as a random draw with no identity of their own – each essay marked by whichever two markers were free, each patient scanned by whoever was on shift – so a condition’s systematic tendency cannot be separated from noise and stays in the error term. It gives the smallest ICC of the three, 0.66 for the spend pair, because the menu effect that the two-way models set aside as an occasion difference here counts as error. It is the model for a design in which the conditions are not the same set for every subject; where they are, it throws information away.

Two-way random effects model treats the conditions as a fixed set that every subject saw – the same two markers marked every essay, every member was measured in both windows – and as a sample from a larger population of possible raters or occasions, so that the study’s conclusion is meant to generalise to raters like these. It separates the condition effect from the error and charges it against agreement: the spend pair reads 0.67, an absolute-agreement coefficient in which the 1.09 shift costs a little. It is the usual choice for an inter-rater study – the raters are a sample of the raters a clinic could employ – and the app’s default.

Two-way mixed effects model is the same design with the conditions treated as the only ones of interest – these two raters, these two windows, no generalisation intended – and it asks for consistency alone: the condition effect is separated out and then ignored, so that a constant offset costs nothing. The spend pair reads 0.70, the highest of the three, and it is the coefficient to report when the question is whether the two conditions rank the subjects alike. Since a fixed set of raters with their offsets removed is exactly what a set of questionnaire items is, this model over the items of a scale reproduces the scale’s Cronbach’s alpha in its average-measures form: the three food items give 0.71 and the six items with slow flipped 0.70, alpha’s own values.

Single and average measures is the form: whether the coefficient describes one measurement – one rater’s mark, one window’s spend – or the mean of all $k$ of them. The average-measures ICC is always the higher, by the Spearman–Brown step that a mean of $k$ readings takes over one reading: the spend pair’s single-measures coefficients of 0.66, 0.67 and 0.70 become 0.80, 0.80 and 0.82 for the mean of the two windows. Report single measures when the scores that will be used in practice come from one rater or one occasion – almost always – and average measures only when every future score will be the mean of the same number of raters. The manual’s ICC options are these two choices, model and form.

Agreement on categories #

Chance agreement is how often two raters would give the same category if each were guessing at their own rate – the floor a raw percentage of agreement has to be read against. The owner’s rule and the members’ own answers agree on 143 of the 193 members who gave an overall rating, 74%, which sounds respectable; but 68% of those members say they would recommend and the rule says so of 48%, and two raters saying yes that often would agree by chance on 0.68 × 0.48 + 0.32 × 0.52 = 49% of members. Two thirds of the observed agreement is what guessing would have produced, and a category that one rater gives 90% of the time makes chance agreement so high that 85% observed agreement can be worse than a coin toss. Every coefficient in this section is a way of subtracting that floor.

Cohen’s kappa is observed agreement between two raters with chance agreement taken out, scaled so that 1 is perfect agreement and 0 is agreement no better than chance: with $p_o$ the observed proportion agreeing and $p_e$ the chance proportion,

For the rule against recommend, κ = (0.74 − 0.49) ⁄ (1 − 0.49) = 0.49: the two classifiers agree about half as well as they could beyond chance. The app reads kappa on the Landis and Koch bands – below 0 poor, to 0.20 slight, to 0.40 fair, to 0.60 moderate, to 0.80 substantial, above 0.80 almost perfect – so 0.49 is moderate, and the card prints it as Cohen’s κ with a bootstrap interval and a p-value against zero. A negative kappa means the raters agree less than chance, which usually means one of them has the categories the other way round. Kappa is for two raters and unordered categories; the three coefficients below extend it.

Weighted kappa is kappa for ordered categories, where a disagreement by one step should cost less than a disagreement by four: each pair of ratings is weighted by the square of its distance, so that a taste of 4 against a fresh of 5 counts as nearly agreeing and 1 against 5 as full disagreement. Read as two raters of one member’s food, taste and fresh give the same rating for 41.5% of members and ratings within one point for 81.5%; unweighted kappa sees only the 41.5% and reads 0.23, quadratic-weighted kappa credits the near misses and reads 0.51. The app applies the quadratic weights whenever the score variable is typed ordinal and prints Weighted Cohen’s κ; with those weights kappa comes close to the consistency ICC on the same ratings, which is why it is the version to set beside one.

Light’s kappa is kappa for three or more raters on ordered categories: quadratic-weighted Cohen’s kappa for every pair of raters, averaged. The three food items as three raters of one member’s food give pairwise kappas of 0.51 for taste with fresh, 0.46 for taste with value and 0.37 for fresh with value, and Light’s kappa of 0.45. It has no exact null distribution, so the app prints it with a bootstrap interval and no p-value.

Fleiss’ kappa is kappa for three or more raters on unordered categories, with chance agreement computed from how often each category was used overall rather than rater by rater – so it also fits a design in which the raters differ from subject to subject. Score each food item as high (a 4 or a 5) or not and read the three as three raters of whether the member liked the food: they are unanimous on 53% of members, and Fleiss’ kappa is 0.37 – fair, on the same bands – with fresh calling 58% of members high, taste 53% and value 46%. The app picks the extension by the score variable’s type: ordinal takes Light’s, categorical Fleiss’.

Krippendorff’s alpha (α) is a chance-corrected agreement coefficient built to work where kappa does not: any number of raters, missing ratings – a subject rated by only some of the raters still contributes – and any level of measurement, nominal, ordinal or interval, through a distance function that says how far apart two ratings are. It is one minus observed disagreement over expected disagreement, so on two raters and nominal data it is nearly kappa: 0.47 for the rule against recommend, beside κ = 0.49. On interval data it behaves like an absolute-agreement ICC – 0.65 for the spend pair, beside ICC(2,1) = 0.67 – and, like it, it charges systematic offsets: the six items as raters, slow unflipped, score only 0.07, since the items have different means and one of them runs backwards. Its author’s bands are stricter than either of the other scales – below 0.667 unacceptable, to 0.800 tentative, above it acceptable – and the app applies them, so 0.47 is unacceptable where κ = 0.49 was moderate. The card’s Interpretation column grades one coefficient per row, the first of ICC, kappa, W, α, r and ρ that the row carries, so the verdict beside a row with both κ and α is kappa’s, and α’s is read from its own bands. Its interval is bootstrapped and it has no p-value.

Kendall’s coefficient of concordance – W – is agreement among three or more raters about the order of the subjects, from 0 (the raters’ rankings are unrelated) to 1 (every rater ranks the subjects identically), a rank-based cousin of the consistency ICC. The six items as six raters of each member’s satisfaction, as recorded, agree to W = 0.26; with slow reverse-scored first, so that all six run the same way, W rises to 0.40, and the three food items alone reach 0.63. The app reads W on Schmidt’s (1997) bands, the ones published for the coefficient itself – very weak below 0.3, weak from 0.3, moderate from 0.5, strong from 0.7 and unusually strong from 0.9 – so the three read very weak, weak and moderate: six five-point items rank two hundred members with many ties, and a rater who ranks backwards pulls the coefficient down as surely as it pulls alpha down. Two conditions give W = (1 + ρ) ⁄ 2 exactly, Spearman’s ρ restated, so the app reports it only from three. The effect sizes page describes the same coefficient with the roles turned round – the members as raters agreeing about the order of the six items, the Friedman test’s companion – and there it is 0.02 on the same bands: the same data, a different question, and W answers whichever way the table is turned.

How much can one score wobble? #

Within-subject standard deviation is the standard deviation of one subject’s repeated measurements around that subject’s own mean, pooled across subjects: how far a single reading typically lands from what the subject would average over many readings, in the score’s own units. It is the standard error of measurement estimated from repeated measurements rather than from a reliability coefficient, and it is what the module’s SEM & SDC computes, from the error term of the analysis of variance that matches the chosen ICC model. For the spend pair it is 2.16 under the two-way random model and 2.02 under the two-way mixed: the mixed model’s error is a member’s wobble once the 1.09 shift has been set aside, and the random model adds the shift back in, since in the absolute-agreement sense it too is a way the two readings fail to agree. Against member spends that average 8.66 before and 9.75 after, a single figure is uncertain by about two units either way. Here the number is an upper bound on the till’s noise, because a member’s genuine response to the new menu is in it as well; a test–retest design in which nothing changed between the occasions would give the measurement’s error alone.

Smallest detectable change is the smallest difference between two measurements of one subject that is larger than measurement error could plausibly produce – the change a study can call real in one person. Two readings each carry the within-subject SD, so their difference carries $\sqrt{2}$ times it, and at 95% confidence

For the spend pair that is 5.98 units under the two-way random model and 5.60 under the mixed: a member whose average spend moved by 3 units between the windows has not demonstrably changed, one who moved by 7 has. It is the number a clinician needs to say a patient has improved and a teacher to say a pupil has progressed, and a measure whose SDC is a large share of its range – 6 units where members’ spends run from 1.5 to 20 – can track groups but barely individuals. Under the mixed model the SDC is exactly the half-width of the limits of agreement below, 5.60: the two are one idea in two vocabularies, and the difference between 5.98 and 5.60 is the shift, which the absolute-agreement SDC folds in and Bland–Altman reports separately as the bias. The literature also calls it the minimal detectable change (MDC) and, for a group mean, the smallest real difference.

Bland–Altman: how far apart are two measurements? #

Bland–Altman plot draws one point per subject, the mean of the two measurements along the bottom and their difference up the side, so that the eye can see three things a coefficient hides: whether the differences sit around zero or off it, how widely they scatter, and whether the scatter changes with the size of what is measured. For the spend pair the 199 points – one member’s after-figure was the catering order and is blank – sit around −1.09, the difference taken as before minus after in the order the conditions appear in the data, and they scatter about as widely for members who spend 5 as for those who spend 15. Three horizontal lines summarise them, the bias and the two limits of agreement, and a fourth, the slope, tests the funnel. Bland and Altman proposed the plot for comparing two methods of measuring one quantity – a new blood-pressure device against the old – where the question is whether the new can replace the old, and that is the reading the vocabulary carries; the café has no such pair, so the plot here compares two occasions, and its bias is a real change rather than a method’s error.

Systematic difference is the mean of the differences – how much one condition reads higher than the other on average – and the Bland–Altman card calls it the bias. For the spend pair it is −1.09 units, before minus after: members spent 1.09 more per visit after the menu change, with a confidence interval of [−1.49, −0.69] that keeps clear of zero. In a method-comparison study the bias is the new device’s calibration error, and a bias whose interval excludes zero says one method reads systematically high; here it is the menu effect, the same quantity the paired comparison tests – and the card by design offers no test of it, since whether 1.09 units matters is a question about units of spend, not about a p-value. It is the same offset that the absolute-agreement ICC charged against the spend pair and the consistency ICC ignored.

Limits of agreement are the two lines between which most individual differences fall: the bias plus and minus 1.96 standard deviations of the differences, so that at the default 95% level about 95 subjects in 100 have a difference inside them. The spend pair’s differences have a standard deviation of 2.86, so the limits are −1.09 ± 1.96 × 2.86 = −6.69 and +4.51: a member’s after-spend is typically anywhere from 6.7 units above their before-spend to 4.5 units below it, and 190 of the 199 members fall inside – 95.5%. They are the answer to the method-comparison question in the measurement’s own units: if two thermometers’ limits are ±0.3 degrees, a clinician decides whether 0.3 degrees is tolerable, and no coefficient can make that decision. The card prints a confidence interval beside each limit, and it answers a different question – how precisely the limit itself has been estimated from this sample: the lower limit here is −6.69 with an interval of [−7.37, −6.00]. A wide gap between the limits says the two conditions disagree, and more subjects will not narrow it; a wide interval on a limit says the sample is small, and more subjects will. A study reports both, and a reader who confuses them takes the sample’s precision for the methods’ agreement. The limits assume the differences are roughly normal and scatter evenly across the range – the plot is where both are checked.

Proportional bias is disagreement that grows with the size of what is measured – two scales that agree on light parcels and drift apart on heavy ones, so that a single pair of limits is too wide at one end of the plot and too narrow at the other. The card tests it by regressing each subject’s difference on the pair’s mean and printing the Slope with its interval: −0.04 [−0.16, 0.08] for the spend pair, an interval that includes zero, so the menu effect is much the same for big spenders and small and the flat limits describe everyone. An interval that excludes zero is the funnel, and the remedy is to analyse the logarithms, so that the bias and limits become ratios – one condition reads 4% higher – which the module’s Analyse on the log scale does.

Sensitivity and bias #

Signal detection reads two binary conditions as a true state and an observer’s response to it, and crosses them into a 2 × 2 table: a hit is a yes to a case that was a yes, a miss a no to it, a false alarm a yes to a case that was a no, and a correct rejection a no to it. Take the members’ own recommend answers as the truth and the owner’s overall ≥ 8 rule as an observer trying to predict them: of the 131 members who would recommend, the rule catches 87 and misses 44; of the 62 who would not, it wrongly flags 6 and correctly passes over 56. The framework’s point is that a rater can agree poorly with the truth for two quite different reasons – because it cannot tell the cases apart, or because it can but sets its threshold too high or too low – and a single agreement coefficient blends them: kappa reads 0.49 for this table and cannot say which. Which condition is the truth is the analyst’s decision, since swapping the roles changes every number below; which of a variable’s two levels counts as the signal the module decides itself, taking the higher – yes over no, 1 over 0 – and naming it on the card.

Hit rate and false-alarm rate are the two numbers every index is built from: the hit rate is hits over all true yeses, 87 ⁄ 131 = 0.66, and the false-alarm rate is false alarms over all true noes, 6 ⁄ 62 = 0.10. A classifier’s sensitivity is the hit rate under another name and its specificity is one minus the false-alarm rate – the owner’s rule is the very cut the regression page’s model reaches at its threshold, and gives the same 0.66 and 0.90. A good observer has a high hit rate and a low false-alarm rate; the trouble is that saying yes more often raises both, so neither rate alone measures skill, and the indices below combine them into one number for skill and another for the threshold.

d-prime – d′ – is the sensitivity index: the distance between the yes cases and the no cases on the observer’s evidence scale, in standard deviation units, computed as the z-score of the hit rate minus the z-score of the false-alarm rate. Zero is an observer who cannot tell the two apart – its hit rate equals its false-alarm rate – and each unit is one standard deviation of separation; 1.68 for the owner’s rule says the members who would recommend and those who would not sit about one and two-thirds standard deviations apart on the overall scale the rule reads, which is decent discrimination. A rate of exactly 0 or 1 has no finite z, which is what the module’s Extreme rates correction is for – its default adds half a case to every cell before the rates are taken, so 87 ⁄ 131 enters as 87.5 ⁄ 132. The card’s A′ is the distribution-free companion, on a scale from 0.5 (chance) to 1, defined even at those extremes – 0.87 here.

Response bias is where the observer puts its threshold – how much evidence it wants before saying yes – as distinct from how well it discriminates. The index c is the threshold’s distance from the neutral point midway between the two distributions, in the same standard-deviation units as d′: zero is an observer with no lean, above zero a conservative one who says yes too rarely, below zero a liberal one who says yes too readily. The owner’s rule reads c = 0.42: it calls 48% of members recommenders where 68% say they are, so it is conservative – it misses 44 members who would recommend to avoid flagging 6 who would not. β says the same on a ratio scale, 2.03 where 1 is neutral, and B″ on a scale from −1 to 1, 0.44 where 0 is. Separating the two indices matters because they answer to different remedies: a low d′ says the rule reads the wrong evidence and no threshold will fix it, while a large c says the evidence is fine and the threshold is misplaced. Lower the owner’s bar to 7 and the rule says yes to 69% of members, almost the members’ own 68%, and its bias flips to liberal, c = −0.33, with a hit rate of 0.84 and a false-alarm rate of 0.37 – and d′ drops to 1.31, since a 0-to-10 rating is not quite the pair of equal normal distributions under which d′ would be the same at every threshold. Report a sensitivity index and a bias index together; either alone describes half of the observer.

Reporting agreement #

For an agreement study, name the design – which conditions, how many subjects, whether every subject saw the same conditions – then the coefficient with its model and form and its confidence interval, and, for a continuous measurement, the SEM or the limits of agreement in the measurement’s units, since a coefficient alone does not say whether the disagreement matters: “Spend per visit in the two eight-week windows agreed to ICC(2,1) = 0.67, 95% CI [0.55, 0.76], two-way random, single measures, n = 200 members; the mean difference (before − after) was −1.09 units, 95% CI [−1.49, −0.69], with limits of agreement from −6.69 to 4.51.” For categories, give the kappa variant and the marginal rates the chance agreement was computed from; for a signal-detection reading, which condition was the truth and a sensitivity index beside a bias index. The manual’s checklist has the full list.

Where this appears in DataSuite #

• Reproducibility & agreement – every coefficient on this page: the ICC with its model and form, the kappa family and Krippendorff’s α chosen by the score variable’s type, Kendall’s W from three conditions up, SEM and SDC under the ICC’s model, the Bland–Altman plot and table with their log-scale and repeated-measurement options, and the signal-detection census and indices.
• Reliability analysis – the one-occasion question, internal consistency; the scale reliability page is its concept page, and its SEM is the same quantity as the within-subject SD here, estimated from a coefficient instead of from repeats.
• Comparison analysis – the paired designs that test the systematic difference Bland–Altman reports without testing, and Kendall’s W as the Friedman test’s effect size, with the roles turned round.
• Regression analysis – the classifier’s sensitivity and specificity, which the hit and false-alarm rates restate, and the ROC analysis that walks every threshold.
• Data transformation – the table converter that stacks two columns into the long format this module reads.
• Analysis planner – the attenuation calculator, which takes a test–retest ICC as readily as an alpha.