Reproducibility & agreement – method notes #

The decisions behind the reproducibility & agreement module – which estimator produces each coefficient, how its interval is built, where its bands come from, what a declared design changes and what is refused, and the numbers that validate them, each computed on the example dataset (cafe.csv, its spec beside it). Sections follow the manual’s.

Data layout #

A repeated subject–condition pair is refused rather than pivoted. tidyr::pivot_wider() turns a repeated cell into a list-column, which psych::ICC() consumes without an error and reports as ICC = 0, so a botched merge would print a plausible number. The module counts the repeats itself – in the layout preview, and again when the run is asked for – and refuses every wide-matrix metric while the data hold one; Bland-Altman is admitted only under a declaration that gives the long frame a path of its own (Repeated measurements, below). Rows missing the subject or the condition are dropped R-side before the pivot, because a missing condition would otherwise pivot into an NA rater column of its own and count as a condition – which silently switches the kappa variant and the correlations’ availability – and a row missing its occasion belongs to no pair and leaves the same way.

Reproducibility metrics #

The kappa variant follows the rater count and the score’s type. Two conditions run irr::kappa2(), with weight = "squared" when the score is ordinal and unweighted otherwise; three or more run Light’s κ for an ordinal score – kappa2(weight = "squared") over every pair of conditions, averaged (Light 1971) – and irr::kappam.fleiss() for a categorical one (Fleiss 1971). Quadratic weights are the choice because under them weighted κ approaches the intraclass correlation on the same ratings (Cohen 1968), so an ordinal kappa reads beside an ICC; Fleiss’ extension takes chance agreement from the pooled category frequencies, which is what fits a design where the raters differ from subject to subject, but it has no weighted form, hence Light’s for ordered categories. Cohen’s and Fleiss’ p-values are the packages’ large-sample tests; Light’s κ has no distributional form and prints none.

Krippendorff’s α is irr::kripp.alpha() for the estimate and an inline coincidence-matrix routine for the bootstrap. The distance function is set by the score’s type – nominal for categorical, ordinal for ordinal, interval for continuous – since that is what the type declares about the distance between two ratings. The bootstrap recomputes α on every resample, and kripp.alpha()'s overhead per call would dominate the loop, so the replicates use the module’s own routine: one minus observed over expected disagreement on the coincidence matrix (Krippendorff 2004), with the nominal, interval and ordinal metrics – the ordinal one the squared count of ratings in the categories between two ratings, applied to observed and expected disagreement alike, since weighting only the expected side would misstate the coefficient. A subject rated under fewer than two conditions contributes no pair and drops out, which is why α’s N is the subjects with at least two ratings.

Validation. On the café’s spend pair the inline routine and kripp.alpha(method = "interval") both read 0.6545.

Kendall’s W is withheld at two conditions. irr::kendall() ranks its input, and with two raters the concordance coefficient is exactly $(1+{\rho }_s)/2$ for the Spearman correlation of the two columns – the Spearman row restated on another scale, under a different p – so the R side does not compute it and a note under the tables says why. From three conditions it runs with correct = TRUE, the tie correction, and its p is the package’s chi-square approximation (Kendall & Babington Smith 1939). It is graded on Schmidt’s (1997) ladder, the one benchmark table published for W itself – 0.1 very weak, 0.3 weak, 0.5 moderate, 0.7 strong, 0.9 unusually strong agreement, each value the floor of its band, so the lowest runs to 0.3 – rather than on the ICC bands the inter-rater literature often reads W beside, which would call a W of 0.72 moderate where Schmidt calls it strong; the comparison module’s Friedman companion reads the same coefficient through the same band function, so one W meets one verdict whichever module reports it.

Validation. On the café’s spend pair Spearman’s ρ is 0.679, and kendall() on the two columns returns W = 0.839 = (1 + 0.679) ⁄ 2.

Spearman’s interval is a Fisher-z interval with the Bonett–Wright standard error. cor.test(method = "spearman") returns no interval, so the module builds one: ρ is clamped to ±0.9999, Fisher-transformed and given the standard error $\sqrt{(1+{\rho }^2/2)/(n-3)}$ (Bonett & Wright 2000), since the Pearson $1/\sqrt{n-3}$ runs about 10% narrow for a rank correlation; the bounds are transformed back. Pearson’s r keeps cor.test()'s own interval and p. Both correlations are graded on correlation-strength bands – 0.10, 0.30, 0.50, with 0.70 as the common very-strong extension – rather than the agreement scale, because a correlation measures consistency and not agreement, and the band is read on the signed value, so a negative coefficient lands at the bottom.

SEM follows the ICC model’s agreement-or-consistency choice. Weir (2005) and de Vet et al. (2006): the SEM that belongs beside an absolute-agreement ICC includes systematic rater differences, the one beside a consistency ICC excludes them. The module fits the analysis of variance the chosen model implies – aov(score ~ subj) for one-way, aov(score ~ subj + rater) for the two-way models – and takes $\sqrt{{\text{MS}}_{\text{residual}}}$ for one-way and two-way mixed, while two-way random adds the rater variance component, $\sqrt{{\text{MS}}_{\text{residual}}+({\text{MS}}_{\text{rater}}-{\text{MS}}_{\text{residual}})/n}$, floored at the residual. SDC is $\text{SEM}\cdot z\cdot \sqrt{2}$ with $z$ the quantile of the configured confidence level – two measurements each carry the SEM, so their difference carries $\sqrt{2}$ of it – which is why its header carries the level. Its interval is the SEM’s bootstrap interval scaled by the same factor rather than a second resampling, since the two are one estimator.

Validation. On the café’s spend pair the SEM is 2.16 under the two-way random model and 2.02 under the two-way mixed – the two-way random model adds the variance component of the 1.09-unit shift between the two windows, which the mixed model sets aside – and the SDC at 95% is 5.98 against 5.60.

Every coefficient but ICC and the two correlations takes a percentile bootstrap over subjects. Kappa’s large-sample standard error (Fleiss, Cohen & Everitt 1969) breaks down at extreme marginals, Light’s κ and Krippendorff’s α have no closed-form interval at all, and none of the SEM’s or the signal-detection indices’ analytic intervals behaves across the full range of inputs – so all of them share one loop: subjects resampled with replacement, bootstrap replications times under the bootstrap seed, every selected coefficient read off each resample, the interval the plain percentile quantiles (Efron 1979, 1981). An interval is withheld when fewer than half the replicates produced a value: a resample that lands on a constant column or an empty cell yields NA, and quantiles over a minority of survivors would print an interval the estimator mostly could not compute. ICC, Pearson r and Spearman ρ keep their analytic intervals, which need no loop.

ICC options #

The six coefficients are psych::ICC()'s rows, and the default is ICC(2,1). Model and form select the row – ICC1, ICC2 and ICC3 for single measures, ICC1k, ICC2k and ICC3k for average – with the interval at alpha = 1 − confidence level and the p from the row’s F test. Two-way random, single measures is the default because it answers the usual inter-rater question (Koo & Li 2016): the same raters rated every subject, they stand for the raters a study’s conclusion is meant to hold for, and the score that will be used in practice is one rater’s. The six rows are Shrout and Fleiss’s (1979) cases, in which absolute agreement rides with the random-effects models and consistency with the mixed one, so agreement against consistency is not a separate control; McGraw and Wong’s (1996) fuller cross-classification would double the options for a distinction the model already makes.

Reading reproducibility results #

Each coefficient is graded on the bands its own literature established, and one coefficient per row. ICC on Koo and Li’s (2016) 0.50 ⁄ 0.75 ⁄ 0.90, kappa on Landis and Koch’s (1977) six bands, Kendall’s W on Schmidt’s (1997) 0.3 ⁄ 0.5 ⁄ 0.7 ⁄ 0.9, Krippendorff’s α on its author’s 0.667 and 0.800 (Krippendorff 2004), the correlations on correlation-strength bands – applying the ICC bands to α would call 0.70 moderate where its author allows only tentative conclusions, and applying them to r would grade a consistency coefficient as an agreement one. A row carries several coefficients of the same agreement, so the verdict grades one: the first present of ICC, κ, W, α, r and ρ, agreement coefficients ahead of consistency ones, and the cell names which, so that a row with both κ and α is read as kappa’s verdict and α is read against its own bands from the manual’s table. The p column is printed only where a metric has a finite p in some row – Light’s κ, SEM, SDC and α never do – to avoid a column of dashes.

Bland-Altman agreement #

The limits’ intervals use Bland and Altman’s (1999) variance, on t. A limit is $\bar{d}\pm z\cdot s$, an estimate from two estimates, and its variance is $\mathrm{Var}(\bar{d})+z^2\mathrm{Var}(s)$ with $\mathrm{Var}(s)=s^2/(2\text{df})$ – the 1999 expression $s^2(1/n+z^2/(2(n-1)))$ in the one-measurement design – so each limit carries an interval built on the t quantile at the differences’ degrees of freedom, and the bias its own on $s/\sqrt{n}$. The expression is written with the degrees of freedom as a term rather than as $n-1$, so that the clustered design (Repeated measurements, below) can pass its own. Three pairs is the floor: below it the estimates are returned empty with the pair count.

Validation. On the café’s spend pair – 199 complete pairs, a bias of −1.09 and a standard deviation of the differences of 2.86 – the limits are −6.69 and 4.51, and the lower one’s interval is [−7.37, −6.00].

The bias carries no test against zero. Bland and Altman’s (1986) argument is that the question in a method comparison is whether the disagreement is small enough to be tolerated, which is a judgement about the width of the limits in the measurement’s units; a p-value on the bias answers whether the mean difference is detectably non-zero, which a large sample makes true of any trivial offset and a small one false of a serious one. The card therefore prints the bias with its interval and no p, and the same reasoning keeps the proportional-bias slope as an interval rather than a test.

The proportional-bias regression always runs. Whether one constant pair of limits describes the data is not something the reader should have to ask for, so every comparison regresses the differences on the pair means (lm(d ~ m), from three pairs) and prints the slope with confint()'s interval – the cluster-robust one under an occasion design – drawing the fitted line on the plot and warning under the table when an interval excludes zero (Bland & Altman 1999).

The SDC and the half-width of the limits coincide under the mixed model. With two conditions the residual mean square of the two-way analysis of variance is exactly half the variance of the paired differences, so $\text{SEM}\cdot \sqrt{2}=s_d$ and the SDC at a given level is $z\cdot s_d$ – the half-width of the limits of agreement at the same level. The two absolute-agreement models add the between-condition component to the SEM, so their SDC comes out larger: they fold the systematic difference into the interval where Bland-Altman reports it separately as the bias.

Validation. On the café’s spend pair the limits are −6.69 and 4.51, a half-width of 5.60; the SDC at 95% is 5.60 under the two-way mixed model and 5.98 under the two-way random.

Every pair is reported past two conditions, and its plots are built on first open. Bias and limits are defined on a pair and do not generalise to k conditions, so k(k−1)/2 comparisons per variable are enumerated, each naming its pair and the direction of subtraction, rather than one arbitrary pair or none. The plots of one variable then collapse behind a <details> summary and are drawn the first time it is opened: a plot built inside a closed <details> has no layout, so the chart scaffold’s viewBox fit polls a zero-size box until it gives up and ships the plot clipped, and deferring also keeps k(k−1)/2 plots per variable off the initial paint.

Analysing on the log scale #

The transform is applied where the measurements are read, so a repeated design averages logs rather than logging an average. Logging both measurements turns a scatter that grows with magnitude into a constant one and the difference into a log ratio (Bland & Altman 1999), and everything downstream then runs in log units: the cell means of a repeated design are means of logs, the bias and the limits are exponentiated for the table – a geometric mean ratio and ratio limits, each interval exponentiated with its estimate – while the slope, a coefficient of the log difference on the log mean, is dimensionless and passes through. The plot keeps the log units so that its lines and its axis agree with each other, and a note names the split so that the table and the picture are not read as disagreeing numbers. A measurement that is not positive has no logarithm and is set missing before anything runs, counted in pairs – the unit the table’s N is in – so the reported n never shrinks silently.

Repeated measurements #

Two declarations, because the data cannot tell the designs apart. Bland and Altman (2007) give two estimators for repeated measurements, and the duplicated rows look identical under both: whether two readings taken in one sitting belong together or merely repeat is a fact about the measurement protocol, so the module asks rather than guesses, and refuses Bland-Altman on repeated rows until it is told. Picking wrong is not a rounding error – summarising paired repeats forces genuine between-occasion movement of the true value into within-subject noise, and pairing unpaired repeats invents an ordering. With the repeats declared an unordered bag, the cells are summarised to their means and each condition’s within-subject variance is pooled as the residual mean square of replicates on subject, which stays unbiased when replicate counts are unbalanced; Limits for the average of repeated measurements stops there, and Limits for a single measurement adds each condition’s within-subject variance back to the variance of the cell-mean differences, net of the sampling noise those means already carry – exact under unbalanced counts, and equal to the published $(1-1/m)$ form when every subject carries the same m. The between-subject component is floored at zero, since where subjects barely differ the means’ noise can exceed the variance observed between them.

The occasion design changes the intervals, not the estimates. Naming an occasion pairs replicate j of one condition with replicate j of the other and lets the true value move between occasions – the second 2007 design – so a subject contributes several differences, and those are not independent. The bias and the limits are the ordinary mean and $\bar{d}\pm z\cdot s$ over all pairs; what changes is where the variance of a single difference comes from and how much each estimate is worth. One-way random-effects components of the differences by subject supply it – their sum is the variance the limits describe, and it is what the observed variance stops estimating once subjects contribute several differences – the mean’s variance is exact under unequal cluster sizes, and Satterthwaite’s approximation gives the sum the degrees of freedom the limits’ interval reads; when the between-subject component floors at zero the sum is the within component alone, on its own degrees of freedom. The slope’s interval is the CR1 cluster-robust sandwich over subjects on the subjects’ degrees of freedom (Liang & Zeger 1986), since differences from one subject share that subject’s disagreement and the ordinary regression interval is too narrow. A subject with exactly one occasion is not clustered, so a dataset where every subject has one gives numbers identical to the same run with no occasion named – the label changes nothing on its own – and a subject–occasion cell holding two measurements of one condition has no ordering left to pair on, so it is averaged and counted in a warning. The table reports Pairs and Subjects side by side because only here are they different numbers.

Signal detection theory #

The higher of a variable’s two values is the signal, and the true state is matched by level name. Both roles carry the same two values, so one census settles which of them is “present”: the higher – 1 over 0, yes over no in sorting order – is taken, which is right for the usual codings and costs no control, because relabelling both roles leaves d′ and A′ untouched and only flips the sign of the bias indices; the card names the level so that the sign can be read. Swapping truth against response, by contrast, transposes the table and changes every rate, so that choice is the user’s, and it reaches R as the level’s name rather than a column index because pivot_wider() orders its columns by first appearance, which is not an order the user chose. A variable with any number of distinct values but two, or a run with any number of conditions but two, is skipped, and the card says which requirement failed.

d′, c and β read the corrected rates; A′ and B″ read the observed ones. With $H$ and $F$ the hit and false-alarm rates, $d^{\prime }=z(H)-z(F)$, $c=-(z(H)+z(F))/2$ and $\beta =\exp ((z(F{)}^2-z(H{)}^2)/2)$ (Green & Swets 1966) – the distance between the evidence distributions, the criterion’s offset from their midpoint, both in SD units, and the likelihood ratio at the criterion. A rate of exactly 0 or 1 puts $z$ on an asymptote, and Extreme rates is what keeps it off: the log-linear rule (Hautus 1995) adds 0.5 to every count and 1 to every total whether or not a rate is extreme, and is the default because correcting only the extremes biases d′ more than shrinking every rate slightly does; the $1/(2N)$ rule moves only a rate that reached an extreme, N being the trials on that side of the truth; None leaves the three empty. A′ (Pollack & Norman 1964, in Grier’s 1971 computing form) and Donaldson’s (1992) B″ are the distribution-free counterparts and are defined at the extremes, so they read the observed rates whatever the correction – as does the census, which reports what was observed. B″'s sign is flipped when $F>H$ so that conservative stays positive once the hit rate has fallen below the false-alarm rate, and it is undefined when its denominator $H(1-H)+F(1-F)$ is zero – both rates at 0 or 1, which is every response correct, every one wrong, or every response the same.

Validation. On the café’s recommend against the owner’s overall ≥ 8 rule – 87 hits, 44 misses, 6 false alarms and 56 correct rejections – the log-linear rates 87.5 ⁄ 132 and 6.5 ⁄ 63 give d′ = 1.68, c = 0.42 and β = 2.03, and the observed rates 0.664 and 0.097 give A′ = 0.87 and B″ = 0.44; the hand computation and the card agree.

Missing data #

Each estimator drops a missing rating its own way, so a table’s N is read per metric. A gap in the rating matrix – a subject with no row under one condition, or a score left blank under pairwise deletion – reaches every estimator, and they disagree on what to do with it: irr::kappa2(), kappam.fleiss() and kendall() na.omit() their input and cor.test() keeps the complete pairs, so those run on the subjects rated under every condition; kripp.alpha() reads every pair of ratings a subject holds, so a subject with one rating contributes nothing and one with two of three still counts; psych::ICC() fits its variance components with lme4::lmer() on the long frame and the SEM’s aov() runs on the ratings present, so both use every rating and count every subject with at least one; Bland-Altman and the signal-detection census use the complete pairs of the two conditions they compare. Dropping the gaps in the module instead – listwise across the selected metrics – would discard ratings that ICC and α can use, which is what the listwise setting is for when one sample is wanted; so each metric carries the count it ran on, and the tables share one N column only while every metric in every row agrees, splitting into one per metric as soon as one differs, since a shared header right for most cells is a wrong number in the rest.

R reproducibility #

The citation list follows what ran, not the repertoire. Each reference is emitted by the code path that produced a result: tidyr always; irr when kappa, Krippendorff’s α or a Kendall’s W that actually ran needed it – W at two conditions is withheld, so neither it nor the package is credited for a column no card shows; psych with lme4 and its paper when an ICC was computed on a continuous score. The kappa variant is cited by what the data selected – Cohen’s for a two-condition run or any ordinal score, since Light’s κ is a mean of Cohen’s and credits both, Light’s and Fleiss’ from three conditions for ordinal and categorical scores – the correlations only at two conditions, and the SEM, Bland-Altman and Krippendorff’s α only when a score of their type was present. Bland and Altman’s 2007 paper is cited by the design that ran, the single-measurement policy and the occasion pairing, since only those reach its variance components, while the averaging policy runs the ordinary estimator on the cell means and adds no reference; the cluster-robust one only under an occasion design, the one that clusters. The signal-detection chain – Green and Swets, A′, B″ – is cited when a variable qualified, the log-linear correction when it was selected, and the $1/(2N)$ rule books nothing of its own, since Macmillan and Creelman already ride the chain; the percentile bootstrap when an interval was bootstrapped. The list can therefore be pasted as that run’s references and nothing else.