English
РусскийReliability analysis – method notes
The decisions behind the reliability analysis module – which estimator produces each coefficient, how its interval is built, where its thresholds come from, what happens when a fit fails, and the numbers that validate them, each computed on the example dataset (cafe.csv, its spec beside it). Sections follow the manual’s.
Reverse-scored items
Reversal mirrors around the declared range, falling back per side to the range observed across the whole scale. A reversed item becomes lowest + highest − old. If the user types both endpoints into Response scale range those are used; a blank side takes the extreme observed across all selected items, not the item’s own. Reversing each item around its own observed range would put differently-answered items on different scales – an item nobody scored 1 on would be mirrored around 2–5 and shifted a point against the rest – and the shift would surface as a spurious item-mean difference. Pooling keeps items sharing one response format comparable when the endpoints are not declared; declaring them is still the right thing to do, since the pooled range can also fall short of the questionnaire’s when no member used an endpoint on any item.
Items keyed against the scale are surfaced, never auto-reversed. psych::alpha() defaults to check.keys = TRUE, which silently flips any item that correlates negatively with the total. The module passes check.keys = FALSE to every psych call and instead runs the keying detection on the scored data – after the reverse-scored list is applied – reporting any item it still reads as keyed against the rest in a note under the metrics table and as a Mark this item for reverse scoring action on the item’s row. A silent flip would hide the most common scoring error – a reverse-keyed item left unticked – behind a plausible alpha; the note makes the user decide, and the action makes deciding cheap. Detection rather than the raw rest correlation, because on a balanced scale – as many items keyed each way – the raw total cancels and the rest correlations name items almost at random.
Detect is the IRT module’s rule, run per scale. The picker’s Detect button and the note above share one routine with the IRT view – a principal-component start, items flipped until no rest correlation on the rescored total is negative, a 2/√n band for the unclassifiable, the majority’s direction kept – whose reasoning, validation and acquiescence limit are in the IRT method notes. With several scales each is detected on its own items, since an item’s keying is relative to the scale it is read in; an item two scales key differently cannot be ticked once for both – an item’s reversal applies in every scale holding it – so it is left unselected and named.
Validation. On the café’s six satisfaction items with slow unreversed, Detect selects slow, and alpha with it ticked, 0.70286, matches a run on data with slow reversed beforehand; the note names slow only while it is left unticked.
Items without variance are withheld before the fit. psych’s estimators throw on a constant column (“missing values (NAs) in the correlation matrix”), and one such item would take every coefficient in the run down with it. The module drops any item with fewer than two distinct non-missing values after reverse scoring and listwise deletion, lists it in the card’s scale information as excluded for no variance, and runs on the rest; a scale left with fewer than two items is skipped and named.
Reliability metrics
Alpha’s interval is Feldt’s F interval; every other coefficient is bootstrapped. Feldt (1965) gives α’s sampling distribution under compound symmetry as a transformed F with n − 1 and (n − 1)(k − 1) degrees of freedom, so the interval is at the two tail quantiles, computed on the complete cases. It is exact under tau-equivalence, approximate otherwise, instant, and bounded below 1 where the symmetric interval is not. Standardized alpha takes the same form, since raw and standardized α coincide under compound symmetry. None of the other coefficients has a usable closed-form interval, so they share one bootstrap.
The bootstrap is the bias-corrected percentile interval, without acceleration. Reliability coefficients are bounded above by 1 and their sampling distributions are left-skewed, which is exactly where the plain percentile interval undercovers; the BC correction shifts the quantiles by the bias between the point estimate and the median of the replicates. Acceleration (BCa) is left out because its jackknife would cost n further refits per coefficient. Each replicate resamples cases with replacement and refits the four estimators once, reading every selected coefficient off that one refit, so adding a coefficient to a run costs little; the replication count is the bootstrap replications setting and the stream is the bootstrap seed. An interval is left blank when fewer than half the replicates produced a value – a coefficient that fails on most resamples has no interval worth printing. omega() and glb.fa() are refit on every replicate, which is why enabling intervals with those two selected is slow on larger scales.
Omega comes from a bifactor solution sized by the item count. psych::omega() is called with nfactors = max(1, min(3, k ÷ 3)) – one group factor per three items, at most three, maximum-likelihood extraction, flip = FALSE so the module’s own keying stands. Below six items the bifactor model is under-identified, so the call degenerates to the congeneric single-factor solution: ω total is reported and ω hierarchical is not defined, and a note under the table says so. The same factor count is reused for the drop table and every bootstrap replicate, so every ω on the card comes from one estimator and the “if deleted” column compares like with like.
Mean split-half is the Rulon–Flanagan coefficient averaged over 10,000 random splits. psych::splitHalf() computes for each split rather than the Spearman–Brown stepped-up correlation, and draws random splits once the item count makes enumeration impractical (from 17 items up); n.sample is pinned to psych’s default of 10,000 so the point estimate, the drop table and the bootstrap replicates all draw the same number of splits, and the draw runs under the app’s seed so a seeded run reproduces. With an even number of items the mean over all splits is algebraically identical to standardized alpha – averaging Rulon–Flanagan over every split is the same computation from another direction – so selecting both is a consistency check, not independent evidence.
λ2 and λ4 come from the split-half fit, λ6 from alpha(). splitHalf() returns Guttman’s λ2 as lambda2 and the maximum split-half as maxrb, which is λ4; Revelle’s β is its minrb. λ6 is alpha()'s G6(smc), the squared-multiple-correlation form. The three lambdas therefore depend on two different estimators, and a failure of one leaves the other’s lambda in place.
Composite reliability and AVE come from a one-factor maximum-likelihood fit. psych::fa(nfactors = 1, fm = "ml", rotate = "none") supplies the standardized loadings; CR is and AVE is the mean squared loading. On a unidimensional scale CR reproduces McDonald’s ω to the third decimal, because the two are the same coefficient from two parameterisations; the module keeps both because the CFA literature asks for CR by name.
Validation. On the café’s food items CR and ω both read 0.718.
Coefficient H pins communalities below 1. Hancock and Mueller’s (2001) divides by for each item’s communality , which is infinite at a Heywood case (a loading of 1 or more). Communalities are capped at 0.9999 before the sum, so a Heywood item yields a finite, near-1 H rather than an error, and the card’s loadings show where it came from.
GLB fails to NA rather than to an error. psych::glb.fa() iterates a factor solution and does not always converge, most often on very short scales – deleting an item from a three-item scale leaves two, on which it usually fails. The call is wrapped so that a failure records a missing value; the row stays on the card with Failed to converge in place of a number, and the “if deleted” cell is blank rather than the whole column dropped.
Each of the four estimators fails alone. The coefficients come from four separate fits – alpha(), fa(), splitHalf() and omega() – and each is wrapped so that an error records the estimator’s key and returns nothing. Every coefficient is gated on its estimator’s fit: those that depend on a failed one keep their rows with Failed to converge in place of a value, and a note under the table names the estimator and lists what it cost (alpha: α, λ6 and the item-total correlations; the factor analysis: CR, AVE and H; split-half: the mean split-half, λ2, λ4 and β; omega: ω). Nothing silently disappears, and nothing that did not depend on the failure is withheld.
λ4, GLB, β and ω hierarchical carry no adequacy verdict. The 0.50 ⁄ 0.60 ⁄ 0.70 ⁄ 0.80 ⁄ 0.90 bands in the Interpretation column were derived for α and ω as estimates of reliability. λ4 and GLB are bounds on reliability – the best split and the largest value the covariance matrix admits – and both capitalise on chance in small samples with many items; β and ω_h measure general-factor saturation, a different quantity. Reading any of them against the same bands would call a bound “excellent” or a saturation index “poor” for reasons that have nothing to do with the scale. They print an em dash instead, and a note under the table says which coefficients and why.
AVE is read on its own bands, and a low value is not alarmed. AVE answers whether the factor explains more than half of each item’s variance on average, so its bands are the convergent-validity thresholds – below 0.50 poor, 0.50 to 0.70 acceptable, 0.70 and above good – rather than the reliability ones. The 0.50 threshold is strict: values of 0.30 to 0.45 are common for well-established scales, because psychological constructs are broad and no single factor explains most of the variance in every item. When α and ω are solid, a low AVE means a broad construct rather than a bad scale, and the card therefore does not escalate it beyond the column label; the threshold carries its full weight only when discriminant validity is the question.
Output options
Scale-level statistics are computed on complete cases only. Under listwise deletion every case that reaches R is complete, so the choice costs nothing. Under pairwise deletion the upstream filter does not run, and a row summed across items with a missing item would be floor-summed – the missing item counted as zero – biasing the scale mean, SD and variance downward. The module sums only rows complete on every item, reports the count as Cases with complete data when it differs from the number of cases, and says so in a note under the table; item-level statistics use each item’s own answered cases, as psych’s pairwise correlations do.
The item-total columns are psych’s raw.r, r.drop and r.cor. raw.r correlates the item with a total that includes it; r.drop correlates it with the sum of the other items, and is the classical corrected item-total correlation the 0.20 ⁄ 0.30 ⁄ 0.50 discrimination cut-offs were defined for, so the Interpretation column reads it and nothing else; r.cor additionally disattenuates for the rest-scale’s unreliability and runs systematically higher – useful as an estimate of the item’s correlation with the construct, not as a discrimination index, and meaningless when the scale’s reliability is near zero or negative. psych computes it as the item’s column sum over the square root of the scale’s common variance (the correlation matrix with squared multiple correlations on the diagonal), so a mis-keyed scale collapses that denominator and every value in the column – not only those beyond ±1 – is computed on it; the module therefore withholds the whole column, with a note under the table, when any value is undefined or exceeds ±1, rather than blanking or clamping cells that would look valid.
“Deletion would improve” is floored at the displayed precision. Any coefficient that would rise by at least one unit of the last displayed decimal when an item is dropped is named, with the gain; a smaller rise rounds away and cannot be seen, so it is not reported. On most scales at least one item nudges some coefficient upward by a hair, and that is sampling noise, not a finding – which is why the entry is a caution in the Interpretation column rather than an issue, and why the manual’s pitfall on deleting items exists.
The drop table needs at least three items. Deleting an item from a two-item scale leaves one, on which every internal-consistency coefficient is undefined, so Reliability if item deleted is skipped below three items and a note explains why.
Floor and ceiling flags fire above a 15% share at the extreme. Following Terwee et al. (2007), an item is flagged for a possible floor or ceiling effect when more than 15% of its answered cases sit at its observed minimum or maximum. The share is taken over each item’s own non-missing cases and against its observed extremes, so a 1-to-5 item nobody scored 1 on is judged against 2 and 5.
The low-variance flag fires below an SD-to-range ratio of 0.10. An item whose standard deviation is under a tenth of its observed range is nearly constant – on a five-point item, an SD below 0.4 – and cannot discriminate; the flag is a warning, since such an item also depresses every coefficient through its small covariances.
Correlations are product-moment, whatever the items’ declared type. Every coefficient on the card is estimated from the Pearson correlation or covariance matrix, and when every item is typed ordinal a note under the inter-item matrix says so. What the card then reports is the reliability of the summed item scores, not of latent variables assumed behind the response categories. Polychoric-based alternatives exist and give higher numbers; whether they are the better estimate is actively disputed, so the module reports the product-moment estimate and names its basis rather than choosing a side.
Missing data
Listwise deletion runs in the app; pairwise leaves the gaps to psych. Under the listwise setting the module drops every case with a missing value on any selected item before the data reach R, so every estimator sees a complete matrix and every table describes the same cases. Under pairwise, the incomplete rows are passed through: alpha(), fa() and the inter-item matrix work from pairwise-complete correlations, item statistics use each item’s own cases, and the scale-level statistics fall back to complete cases (the note above). Imputation replaces the values before the module sees them. The two settings can give different coefficients on the same file whenever missingness is uneven across items, and the reporting checklist asks which was used for that reason.