Scale reliability #

The café survey asked six questions about satisfaction and means to add three of them into a Food score and three into a Service score. Whether that is allowed – whether three answers measure one thing closely enough that their sum means something, and how much noise the sum still carries – is the question of reliability. This page explains what reliability is, how Cronbach’s alpha estimates it and what alpha assumes, the coefficients that relax those assumptions, how the items are read one at a time, why a reverse-worded item has to be flipped first, how many items a scale needs, and how precise one person’s score is. The numbers come from the café data, prepared as that page describes, and from the reliability analysis module with the three fixes done; slow is reverse-keyed, and on this page it is flipped by ticking it under Reverse-scored items in the module rather than in the data, so the file stays as the codebook describes it.

What reliability is #

Reliability is the share of the spread in a score that is real differences between people rather than noise – a number from 0 to 1, where 1 would mean every point of difference between two members’ Food totals is a difference in how satisfied they are with the food, and 0 would mean the totals are as good as random. The Food scale’s reliability is 0.71: about seven tenths of the variance in the totals is food satisfaction and three tenths is measurement error. Reliability is a property of scores in a sample, not of a questionnaire – the same six questions asked of a different clientele, or in another language, give another number – so it is computed and reported for every dataset a scale is used on. In the app’s roles the scale is the set of items selected in the left panel and the score is their sum; a psychologist: ten anxiety items → items, the anxiety total → the score; a teacher: exam questions → items, the mark → the score; a manager: engagement survey lines → items, engagement → the score.

Measurement error is the part of an answer that would come out differently if the same person answered again in the same state: a mood on the day, a word read two ways, a slip of the pen, a five given where a four was meant. No single question is free of it, and it is the reason a scale has several items – errors that are independent of one another partly cancel when the answers are added, and the sum is steadier than any one answer. The whole apparatus of this page is a set of ways to estimate how much error is left in the sum.

True score is the score a person would average over endlessly many equivalent questionnaires – what they would get if the measurement error averaged out. It cannot be observed; an observed score is the true score plus that occasion’s error, and reliability is the ratio of true-score variance to observed-score variance. Everything below estimates it from the one thing that can be seen: how far the items agree with one another. The framework these ideas belong to is classical test theory; item response theory is the other way of asking the question.

Scale is a set of items whose answers are combined – summed or averaged – into one score that stands for a construct. Food is taste, fresh and value, summed to a total from 3 to 15 or averaged to a mean from 1 to 5. The app uses the word in this sense throughout the reliability module: the Scales matrix assigns items to scales, and a multi-scale card heads each block Scale: Food. The same word names the set of answers an item allows – “a 1-to-5 scale” – and the module says response scale when it means that.

Internal consistency is the kind of reliability that can be estimated from a single administration of the questionnaire: how consistently the items agree with one another. If the three food items are measuring the same satisfaction, a member who gives taste a 5 should tend to give fresh and value high marks too, and the correlations among the items are the evidence. It is one of three ways to ask whether a measurement is reproducible – the others ask whether the same people score alike when measured again (test–retest) or when rated by someone else (inter-rater) – the agreement page’s questions – and both need a second measurement that the café survey does not have. Every coefficient on this page is an internal-consistency coefficient.

Cronbach’s alpha #

Cronbach’s alpha is the most widely reported reliability coefficient: it compares the variance of the total score with the sum of the variances of the items that make it up. If the items measure one thing, members who are high on one are high on the others, the item scores pile up in the same direction, and the total varies more than the items would if they were independent; alpha is that excess, scaled to run from 0 to 1 – with $k$ the number of items, $s_i^2$ each item’s variance and $s_X^2$ the total’s,

The three food items have variances of 1.42, 1.32 and 1.46, summing to 4.20; the total has a variance of 7.98; so α = 1.5 × (1 − 4.20 ⁄ 7.98) = 0.71. The app reports it with an exact confidence interval – 0.71 [0.63, 0.77] for Food and 0.67 [0.58, 0.74] for Service – and reads it against the bands in its Interpretation column: below 0.5 unacceptable, 0.5 to 0.6 poor, 0.6 to 0.7 questionable, 0.7 to 0.8 acceptable – the level usually asked of a scale used to compare groups – 0.8 to 0.9 good, 0.9 and above excellent, and above 0.95 a warning that the items may be near-duplicates. Food just clears 0.70 and Service falls a little short, which for three-item scales is unremarkable – see the Spearman–Brown formula. Alpha assumes tau-equivalence; when that fails it understates the true reliability, which is why the app puts McDonald’s omega beside it.

Standardized alpha is alpha computed on the correlation matrix rather than the covariance matrix – as if every item had first been rescaled to a standard deviation of 1 – and, with $\bar{r}$ the mean inter-item correlation, equals

For Food it is 0.712 against a raw alpha of 0.711: the three items are all answered 1 to 5 with SDs of 1.15 to 1.21, so standardizing them changes nothing. The two part company when the items are on different scales – a 0-to-100 slider summed with 1-to-5 items dominates the raw total and the raw alpha, while the standardized alpha describes the scale one would get by standardizing every item before adding. Report the one that matches how the score is actually formed; the app prints both when alpha is selected.

Tau-equivalence is the assumption that every item measures the construct with the same weight – the same factor loading – and differs from the others only in its error. It is what makes alpha an exact estimate of reliability rather than a lower bound. It is rarely true: the café’s food items have standardized loadings of 0.57 to 0.77 in a confirmatory model, so taste carries more of the construct than value. Where loadings differ, alpha runs below the true reliability, by an amount that grows with the spread of the loadings; here it is small – alpha 0.711 against omega 0.718 – and on a scale with one strong item among weak ones it can be several hundredths. A scale whose items load unequally is called congeneric, and the coefficients in the next section are built for it.

Inter-item correlation is the correlation between two items of a scale, and the raw material every coefficient here is made from. Food’s three are 0.51 (taste with fresh), 0.46 (taste with value) and 0.38 (fresh with value), a mean of 0.45; Service’s, with slow flipped, are 0.43, 0.42 and 0.36, a mean of 0.40. Between about 0.15 and 0.50 is the usual range for items that measure one broad construct without repeating one another; above 0.80 two items are saying the same thing twice; near zero or negative, one of them is measuring something else or is reverse-keyed and has not been flipped. The card’s Scale statistics report the mean and the range, and the Inter-item correlation matrix option prints them all – for the six café items together the range is 0.06 to 0.51, and the matrix shows why: correlations of 0.36 to 0.51 within each trio and 0.06 to 0.28 across them, two blocks rather than one.

Spearman–Brown formula gives the reliability a scale would have with a different number of items, if the new items were as good as the old:

the standardized-alpha formula read as a function of $k$. With Food’s mean inter-item correlation of 0.45, three items give 0.71, five give 0.80, six 0.83 and ten 0.89: reliability grows with length, with diminishing returns, because each added item cancels a little more of the error. The rule cuts both ways – a three-item scale that reaches 0.71 is doing well for its length, and a twenty-item scale that reaches only 0.71 has weak items. The analysis planner’s Reliability by number of items section runs the calculation in both directions: Food would need about five items of the same quality to reach 0.80. The formula’s original job was stepping a half-test’s reliability up to the full test, which is where the split-half methods below come from.

Beyond alpha – the factor-based coefficients #

McDonald’s omega estimates reliability from a factor model instead of from the item variances: fit one factor to the items, take each item’s loading ${\lambda }_i$, and compute

the variance the common factor puts into the total, over the total’s variance. Because the loadings are allowed to differ, omega does not need tau-equivalence, and it is the coefficient to trust when it and alpha disagree. On the café scales they barely do: ω = 0.72 for Food and 0.67 for Service, against alphas of 0.71 and 0.67. It is the same quantity the CFA vocabulary calls composite reliability. Its interval is bootstrapped, so it changes a little from run to run unless a bootstrap seed is set. One caution: omega is a coefficient of the total, not of one construct. Run all six café items as one scale and ω rises to 0.77 while α stays at 0.70 – the factor model found two factors and gave the total credit for both, which is honest about how well the sum of six answers hangs together and silent about whether it measures one thing; omega hierarchical answers that.

Omega hierarchical – ω_h – is the share of total-score variance that a single general factor running through every item accounts for, from a bifactor model in which each item loads on the general factor and on one group factor. The six café items give ω_h = 0.44 against an ω total of 0.77: less than half the variance in the six-item total is “satisfaction with the café” as such, and the rest is food satisfaction and service satisfaction in particular, plus error. A scale that is one thing has ω_h close to ω; a wide gap is the signature of a multidimensional total, and the argument for scoring the subscales separately. The app fits the bifactor model from six items up, one group factor per three items and at most three, so shorter scales report ω total alone. It carries no adequacy verdict on the card, because it measures saturation by a general factor, not reliability. The factor module’s Schmid–Leiman transformation of the same six items – two factors, oblimin – gives the same 0.44 and 0.77, so the two routes to the hierarchy agree, and its card prints the three coefficients below beside them.

Omega hierarchical asymptotic – ω_h ⁄ ω_t – is omega hierarchical divided by omega total: the share of the reliable variance in a total score that is the general factor, where ω_h is its share of all the variance, error included. The two answer different questions – ω_h asks how much of the total is g, ω_h ⁄ ω_t how much of what the total measures dependably is g – and the ratio is the one a total score turns on, since error is not something a subscale would claim for itself. The six café items give 0.44 ⁄ 0.77 = 0.58: of the reliable variance in a six-item satisfaction total, a little over half is satisfaction with the café as such and the rest is food and service in particular. The literature gives it one bar, 0.80, above which a total score can be treated as essentially unidimensional (Rodriguez, Reise and Haviland, 2016), and the app reads it against that alone – reliable variance mostly general or reliable variance substantially group-specific. The name is the value ω_h tends to as a scale is lengthened with items of the same structure, once error has averaged out; the factor module’s Schmid–Leiman card prints it as Omega hierarchical asymptotic (ωh/ωt).

Omega subscale – ω_s – is a subscale’s reliability as a score of its own: the share of the variance in that subscale’s total that is common variance, general factor included, and so what to report when Food is scored and used by itself. From the café’s transformed solution ω_s is 0.72 for the food trio and 0.67 for the service trio – the same numbers the module reports as McDonald’s ω when each trio is run as a scale on its own, which is what ω_s is. It is graded on the reliability ladder every other coefficient uses, because it is a score reliability; the factor module prints one per group factor, Omega subscale F1 (ωs), Omega subscale F2 (ωs) and so on, next to the ω_hs it is read with.

Omega hierarchical subscale – ω_hs – is the subscale’s ω_s with the general factor partialled out: the share of the subscale total’s variance that is its group factor alone, what the subscale measures that the total score does not already carry. The food trio’s is 0.42 and the service trio’s 0.39 against ω_s of 0.72 and 0.67, and the two are read as a pair: ω_s − ω_hs, 0.30 and 0.28 here, is the general factor’s share of each subscale, so each café subscale is a reliable score of which a little more than half is specific to it. A subscale with ω_s = 0.85 and ω_hs = 0.15 is perfectly reliable as a score and almost entirely the general factor showing through – scoring it separately adds nothing the total does not have – and a large ω_hs is what justifies a subscale as a construct of its own. Only ω_s takes a reliability verdict: the usual cutoffs are written for a score, and applied to a residualized quantity they would call a well-behaved subscale defective when its low value is a finding about the general factor’s reach. The factor module prints it as Omega hierarchical subscale F1 (ωhs) and so on, beside each ω_s.

Coefficient H is the reliability the construct would have if the items were combined with their optimal weights – each item weighted by how well it measures the factor – instead of simply summed:

It is never below omega, and the gap between them is what a weighted score would buy: Food’s H is 0.74 against ω 0.72, Service’s 0.68 against 0.67, so weighting would gain two hundredths on these scales and the plain sum loses little. A large gap means one or two items carry most of the construct; the loadings say which. The factor module prints it for a Schmid–Leiman hierarchy too, one H for the general factor and one per group factor: on the six café items 0.58 for g and 0.53 and 0.50 for the food and service groups, each below the 0.70 that marks an adequately defined construct – which is what six items split three ways leave.

Two more coefficients on the same card belong to the validity vocabulary rather than to reliability: composite reliability is omega under its CFA name, and average variance extracted asks how much of each item the factor explains – a different question, with a threshold of its own.

Lower bounds and split halves #

Split-half reliability is the oldest estimate: divide the items into two halves, score each half, correlate the two half-scores, and step the correlation up to what the whole scale would show. Different splits give different values, so the app averages the Rulon–Flanagan coefficient – 1 − variance of the difference between the halves ⁄ variance of the total – over up to ten thousand random splits and reports the Mean split-half. For three items every split is one item against two, and the mean is 0.63 for Food and 0.60 for Service, below alpha because the halves are unequal; with an even number of items the mean over all splits equals standardized alpha exactly – 0.705 for the six café items, both ways. The best and worst single splits are Guttman’s λ4 and Revelle’s β.

Guttman’s lambdas are six lower bounds on reliability that Guttman derived in 1945, each guaranteed not to exceed the true reliability under classical test theory. Alpha is the third of them, λ3; the app reports λ2, λ4 and λ6, the three that add information. A lower bound is read differently from an estimate: it says “reliability is at least this”, and the highest bound is the most informative.

Guttman’s λ2 improves on alpha by replacing the sum of item variances with a term that uses the item covariances more fully, and is never below alpha: 0.714 against 0.711 for Food, 0.717 against 0.703 for the six items together. When the two differ by more than a hundredth or two the items are far from tau-equivalent, and omega will differ from alpha as well.

Guttman’s λ4 is the largest split-half coefficient over every possible division of the items into two halves – the best case where the mean split-half is the average. Food’s is 0.68, the best of its three one-against-two splits; the six items give 0.75, from a split that puts food and service items on both sides so that each half has something in common with the other. With many items it capitalises on chance – among thousands of splits one will be lucky – so the app prints it without a verdict.

Guttman’s λ6 estimates reliability from how well each item is predicted by all the others – its squared multiple correlation – and rewards a scale whose items are each explained by the rest: 0.63 for Food, 0.70 for the six items. It runs below alpha on a short scale, where two items cannot predict a third well, and above it on a long one.

Revelle’s beta – β – is the smallest split-half coefficient over all splits, the mirror of λ4: the worst way to cut the scale in two. It is a lower bound on how much of the total-score variance a single general factor explains, because if the items can be divided into two halves that barely correlate, no one thing runs through all of them. Food’s β is 0.59 against a λ4 of 0.68; the six items together give 0.47 against 0.75, and the wide gap says what the inter-item matrix showed – the worst split is Food against Service, and the six are two scales, not one. It carries no verdict on the card.

Greatest lower bound – GLB – is the largest reliability that any set of item error variances consistent with the observed covariance matrix allows: the highest value a lower bound can reach, found by an iterative fit. It is 0.72 for Food and 0.77 for the six items, matching omega in both cases. It is the most optimistic of the bounds and, like λ4, capitalises on chance in a small sample with many items, so it is read as an upper reference rather than a verdict; the fit does not always converge, and the card says so when it does not.

Reading the items #

Item-total correlation is the correlation between one item’s answers and the total score of the scale it belongs to – how well the item goes along with the whole. Because the item is part of the total it is correlated with, the value is inflated, and more so the fewer the items: Food’s three are 0.78 to 0.83. The card prints it as Item-total r; the corrected version below is the one to read.

Corrected item-total correlation is the correlation between an item and the sum of the other items – the item-rest correlation – so the item cannot inflate its own score. It is the classical index of how well an item discriminates between members high and low on the construct, and the bands the app’s Interpretation column applies are the classical ones: below 0.20 very weak, 0.20 to 0.30 poor, 0.50 and above good; a negative value means the item runs against the rest of the scale, which is almost always a reverse-keyed item left unflipped. Food’s are 0.59, 0.52 and 0.49, Service’s 0.47, 0.52 and 0.46 – every item pulling its weight, none outstanding. The card prints it as Corrected item-total r (item-rest).

Disattenuated item-total correlation is the item-rest correlation corrected for the unreliability of the rest of the scale it was correlated with – an estimate of how the item would correlate with the construct itself, measured without error. It runs higher than the item-rest value by construction (Food’s 0.70, 0.62 and 0.57 against 0.59, 0.52 and 0.49) and is not the number to read the discrimination bands against. The correction divides by a reliability, so it makes no sense when the scale’s reliability is near zero or negative: on the unflipped Service items below the module withholds the whole column, with a note under the table, until the scoring is fixed. The card prints it as Item-total r corrected for overlap and reliability.

Alpha if item deleted is the coefficient recomputed with that item left out – one column per selected coefficient, headed α if deleted, ω if deleted and so on. If a coefficient rises without an item, the item was hurting the scale; if it falls, the item was helping. On the Service items with slow unflipped, alpha is −0.52 and rises to 0.60 when slow is deleted – the signature of a reverse-keyed item. On Food, deleting any item lowers alpha, to 0.55, 0.63 or 0.68, as removing a third of a three-item scale must. It is not a licence to prune: a rise of a hundredth or two is sampling noise, the app names a gain only when the displayed precision can show it, and an item should leave a scale for a substantive reason – ambiguous wording, a different construct – not because a number goes up. The pitfalls on the manual say more.

Ceiling effect is a large share of respondents at the top of an item’s range, and a floor effect the same at the bottom. An item that a third of the members answer with the maximum cannot tell those members apart, however different their satisfaction is, and it contributes less to the scale than its correlations suggest. The app flags either when more than 15% of the answered cases sit at the extreme, and every one of the six café items is flagged for a possible ceiling effect – from 21% of members at the top of value to 33% at the top of friendly – which is what a five-point satisfaction item does among broadly satisfied customers, and a reason the scales’ reliabilities are not higher. A test that everyone passes has a ceiling; a pain scale on which most patients report zero has a floor; a satisfaction item that most customers give top marks to has a ceiling, and the shorter the response scale the sooner it appears.

Reverse scoring #

Reverse-keyed item is an item worded so that agreeing with it means less of the construct: “the service was slow” on a satisfaction scale, “I feel useless” on a self-esteem scale, “I found the lecture hard to follow” on a teaching evaluation. Questionnaires mix them in on purpose, to catch respondents who tick the same column all the way down, and the scoring instructions say which items they are. The café’s slow is one – a 5 is bad service – and the codebook says so.

Reverse scoring flips a reverse-keyed item so that it points the same way as the rest: new = lowest + highest − old, so on a 1-to-5 item a 1 becomes 5, a 2 becomes 4 and a 3 stays. What happens without it is not subtle. The Service items with slow left as recorded give an alpha of −0.52, slow correlates −0.36 and −0.42 with the other two items and −0.46 with their sum, and the card flags it in red; with slow flipped the correlations are 0.36 and 0.42 and alpha is 0.67. A negative alpha, or a negative item-rest correlation, is almost always this and nothing else. In the module the fix is a tick under Reverse-scored items – the item is reversed for the analysis and the data are untouched. Detect proposes the ticks from the data, and the card offers Mark this item for reverse scoring on any item that still reads as keyed against the rest. The IRT module takes the same list of its own and reverses the items for the fit, so the data can stay as answered – which its careless-responding screen needs. To store a reversed copy for a scale score, Data transformation’s reverse-scoring method writes it as a new variable.

Response scale is the set of answers an item allows – 1 to 5 for the café items, 0 to 10 for overall – and reverse scoring mirrors around its ends. The app takes the ends from the answers it sees unless they are typed into Response scale range, and the difference matters when nobody used an endpoint: if no member ever gave slow a 1, the observed range is 2 to 5, mirroring around it turns a 5 into a 2 rather than a 1, and the reversed item is shifted one point against the others. The range is read across the whole scale rather than item by item for the same reason, so items sharing one response format stay comparable. Type the questionnaire’s real endpoints whenever you know them.

How precise is one person’s score? #

Standard error of measurement turns a reliability into the uncertainty of an individual score, in the score’s own units:

Food totals have an SD of 2.83 and a reliability of 0.71, so the SEM is 1.52 points on the 3-to-15 total, or 0.51 on the 1-to-5 mean – a member whose Food total is 10 has a true score somewhere around 10 ± 3 at 95% confidence, and could as easily be a 7 as a 13. Service’s is 1.66. It is the number that says whether two members’ scores are really different and whether one member has really changed, and it is why a scale reliable enough for comparing groups (0.7) is not reliable enough for decisions about individuals (0.9 or more). It is not the standard error of a mean, which shrinks with the sample size; the SEM describes one person and does not. The card’s Scale statistics print it when alpha is selected.

Attenuation is what measurement error does to a correlation: two variables measured with error correlate less than the true quantities do, by a factor of $\sqrt{\text{reliability of one}\times \text{reliability of the other}}$. Food and Service means correlate 0.30 in the café data; divided by $\sqrt{0.71\times 0.67}$, that is a correlation of 0.44 between the constructs – close to the 0.45 the CFA estimates directly, since a latent variable is measured without the error. The analysis planner’s Correlation attenuation section runs the formula either way; the practical lesson is that a modest correlation between two unreliable scales may be a strong one between the things they measure, and that no amount of sample size will recover it.

Reporting reliability #

For every scale a study scores, give the coefficient by name with its confidence interval, the number of items, the sample it was computed on, and which items were reverse-scored: “Food satisfaction was the mean of three items (taste, fresh, value); Cronbach’s α = 0.71, 95% CI [0.63, 0.77], McDonald’s ω = 0.72, n = 200. Service satisfaction was the mean of three items with slow reverse-scored; α = 0.67 [0.58, 0.74], ω = 0.67.” Report omega beside alpha whenever the audience will accept it; name any item removed and why; and when the scale is used to classify individuals, give the SEM. The reliability manual has the checklist.

Where this appears in DataSuite #

• Reliability analysis – alpha and omega on by default, the lower bounds and factor-based coefficients as options, the item analysis with its flags, reverse scoring by tick, and several subscales in one run.
• Structural equation modeling and confirmatory factor analysis – composite reliability and AVE per factor, from the same loadings omega is built on; the latent variables page covers them.
• Factor analysis – ω_h, ω_h ⁄ ω_t and the subscale omegas from a Schmid–Leiman transformation of an oblique solution, coefficient H for the general and the group factors, and the unidimensionality check that decides whether a set of items is one scale or two.
• Analysis planner – the Spearman–Brown calculator for scale length, and the attenuation calculator for what unreliability does to a correlation.
• Questionnaire scoring guide – reverse scoring and total scores as a data-preparation task, before any analysis.