Latent variables and model fit #

Nobody can measure “satisfaction with the food” the way they measure a waiting time. What can be measured is a handful of questions that satisfaction should show through – the food tasted good, the ingredients were fresh, it was good value – and the statistics of this page are about the step from the answers to the thing behind them. It explains what a latent variable is, how factor analysis finds one in a set of items, how many to keep and whether one general factor runs above them, how a confirmatory model tests a structure you already believe in, how the fit of such a model is judged, how a scale is checked for validity and for meaning the same thing in two groups, and how item response theory puts each item and each person on one scale. The numbers come from the café data, prepared as that page describes, with the answers kept as the members gave them. slow is reverse-keyed – “the service was slow”, where a 5 is bad service – and it stays that way in the data: each module is told about it in its own keying control instead, IRT by picking it under Negatively keyed items (Detect finds it) and reliability analysis by ticking it under Reverse-scored items (optional). Factor analysis and CFA have no such control and read slow as answered, so there it loads negatively – as strongly as its neighbours, with the opposite sign.

Measuring what cannot be seen #

Latent variable is a quantity that is not in the data as a column and is inferred from the columns that are: the food satisfaction behind taste, fresh and value, the service satisfaction behind friendly, quick and slow. It is defined by what it does – each of its items rises and falls with it – and it is measured with error, since no three questions catch all of it. Almost every construct in a questionnaire is one: anxiety, engagement, ability, brand loyalty. In the app it is what the factor analysis, SEM and CFA and IRT modules estimate, and the roles it is described in are the same across them: the latent variable, its indicators (the items that measure it), and where there is one, the grouping variable the measurement is compared across. A psychologist: five anxiety items → indicators, anxiety → latent variable; a teacher: exam questions → items, ability → the trait; a manager: engagement survey → indicators, engagement → construct.

Latent is the same word as an adjective – a latent mean, a latent correlation – and means “of the unobserved variable rather than of its items”. In a path diagram a latent variable is drawn as a circle or ellipse.

Observed variable is a column in the data – an item, a spend, an age – as opposed to a latent variable inferred from columns. In a path diagram it is a rectangle. Every observed variable in a measurement model is an indicator of something, and in a structural model some are outcomes or predictors in their own right, like overall below.

Observed is the adjective: an observed correlation is the one between two columns, a latent correlation the one between the variables behind them. The two differ by measurement error, and the difference is one of the reasons to model the latent variable at all – see factor correlation.

Indicator is an observed variable used to measure a latent one: taste is an indicator of Food. A latent variable needs at least three (two, if it is correlated with another) for its model to be identified, and the more good indicators it has the better it is measured. The word is CFA and SEM’s; factor analysis and IRT say item.

Item is a question on a questionnaire or a test, and the column that holds its answers. The café’s six items are answered 1 to 5; a test item might be scored right or wrong. Item response theory models items one at a time, and reliability analysis and factor analysis take a set of them as a scale.

Construct is the concept a scale is built to measure – the thing the latent variable stands for in words. “Food satisfaction” is the construct; Food, the factor, is its statistical stand-in; and whether the factor really is the construct is a question of validity, not of fit – see convergent validity.

Factor is a latent variable as factor analysis names it: a common cause of a set of items, estimated from the correlations among them. In an exploratory analysis the factors are found by the algorithm and numbered (F1, F2); in a confirmatory one they are named by the analyst (Food, Service). A factor is not a sum of its items – it is what the items share, with each item’s own noise left out.

Component is what principal component analysis produces in the place of a factor: a weighted sum of the items chosen to carry as much of their total variance as possible. It is a summary, not a cause, and the difference between the two is the subject of PCA.

Measurement model is the part of a model that says which indicators measure which latent variables – Food =~ taste + fresh + value in the app’s syntax, read “Food is measured by taste, fresh and value”. A CFA is a measurement model and nothing else; an SEM adds a structural model on top.

Dimension is one latent variable in a set of items that may contain several. The six café items are two-dimensional: Food and Service. A scale is unidimensional when one latent variable accounts for the correlations among its items, and most of what is done with a scale – a total score, a reliability coefficient, an IRT model – assumes that it is.

Unidimensionality is the assumption that a set of items measures one thing. It is checked before it is relied on: an exploratory factor analysis that wants one factor, or in IRT the ratio of the first to the second eigenvalue of the item correlations – the six café items give 2.44 to 1.31, a ratio of 1.9 where a one-dimensional scale gives 3 or more. The check is what tells the reader to model Food and Service separately rather than as one “satisfaction”.

Path diagram is the picture of a model: ellipses for latent variables, rectangles for observed ones, single-headed arrows for “measures” or “predicts”, double-headed arrows for “correlates with”, and the estimates written on the arrows. The app draws one for every factor, CFA and SEM model when the option is checked, and it is the fastest way to see what a model claims.

Exploring a structure: factor analysis #

Exploratory factor analysis starts with the correlations among the items and asks how many latent variables, loading on which items, would produce them. The café’s six items correlate 0.38 to 0.51 within the food trio and 0.36 to 0.43 in size within the service trio – slow’s two negative, since it runs the other way – but only 0.07 to 0.28 in size across the two – the pattern of two clusters that two factors reproduce.

Exploratory factor analysis finds the factors in a set of items without being told what they are. The analyst chooses how many to extract, an extraction method and a rotation; the algorithm chooses which items load on which factor. On the six café items with two factors, minimum-residual extraction and an oblimin rotation, the answer is the one the survey was written to: taste, fresh and value load 0.76, 0.59 and 0.64 on the first factor and near zero on the second; friendly, quick and slow load 0.58, 0.72 and −0.61 on the second and near zero on the first, slow negatively because it is read as answered. The factor analysis manual walks the three steps – settings, number of factors, the full analysis.

Principal component analysis is the same set of steps with a different aim: not to find the causes of the correlations but to compress the items into a few sums that keep as much of their total variance as possible. On the café items two components reproduce 62.5% of the total variance where the two factors account for 44.1% of it – the components are larger because they carry each item’s own noise as well as what the items share, so the component loadings (0.81, 0.73, 0.81 for the food items) are bigger than the factor loadings (0.76, 0.59, 0.64). Use components to reduce a large set of measurements to a few scores; use factors to measure a construct. The app’s factor module offers both under one extraction method setting, and its default is PCA.

Correlation matrix is what a factor analysis is fitted to – the correlations among the items, not the items themselves – which is why two datasets with the same correlations give the same factors. For answers on a 1–5 scale the app can compute polychoric correlations instead of Pearson ones, which treat the five answers as cut points on a continuous scale; on the café items the Pearson ones are used throughout this page. The choice moves little on clean structure: Spearman correlations shift no café loading by more than 0.06 and read φ = .40, and polychoric ones, with the six items typed ordinal, load 0.79, 0.61 and 0.69 on Food and 0.59, 0.73 and −0.66 on Service at φ = .39 – the same structure as Pearson’s with more common variance, 48.2% against 44.1%.

Common factor model is the model every extraction but PCA fits: each item is a weighted sum of the factors plus a part that is its own, the weights being its loadings and the leftover its uniqueness. Under the model two items correlate only through the factors they share, so the model is fitted to the correlation matrix and never to the items themselves – an extraction looks for loadings whose implied correlations come as close as they can to the observed ones, and how much of each item is common, its communality, is estimated on the way rather than assumed. Two factors on the six café items reproduce all fifteen correlations to within 0.03, and an EFA’s fit indices are summaries of that gap. It is also why a factor analysis says nothing about an individual member until factor scores are estimated afterwards, and why PCA, which keeps the 1s on the diagonal and asks no question about what is common, is a different model rather than a different method.

Reduced correlation matrix is the correlation matrix with each item’s communality in place of the 1 on its diagonal – at the start, the item’s squared multiple correlation – so that only the variance the items share is left to be factored. It is the matrix an EFA extraction works on, and its eigenvalues are what the app’s scree plot draws as the blue series under a factor method, with the plain correlation matrix’s as the grey reference curve. The two series tell different stories: the café items’ reduced eigenvalues run 1.73, 0.58, −0.09, −0.15, −0.19 and −0.20 against the correlation matrix’s 2.44, 1.31, 0.66, 0.58, 0.54 and 0.47. A negative tail is routine – the starting communalities underestimate the true ones – and it is why the Kaiser criterion is read on the grey series: on the blue one it would keep a single factor.

Squared multiple correlation is the R² of one item regressed on all the others – the share of it the rest of the set can predict – and the usual first guess at its communality, which the extraction then refines. On the café items it runs from 0.23 (slow) to 0.36 (taste) against final communalities of 0.36 to 0.60: the guess is always low, because the other items stand in for a factor only as well as they measure it. A value at or above 0.99 says an item is almost a copy of the others put together – a total score left among its parts, an item entered twice – and the app names such an item in its suitability checks, since it makes a Heywood case all but certain.

Ledermann bound is the largest number of factors the common factor model can have on a given number of items before it has more unknowns than the correlations can settle – the largest count at which the model’s degrees of freedom stay at or above zero. Six items allow three factors, and at three the df is exactly 0: the model is saturated, reproduces the correlations perfectly by construction, and has no fit to report. Ten items allow six, twenty allow fourteen. Past the bound the model is not identified, so the app’s Step 2 stops there – the café’s six items are offered one to three factors – and a perfect fit at the bound is no evidence for that many factors, since any data would have given it. PCA has no such bound: a component model estimates no uniquenesses and can take as many components as there are items.

Extraction method is the rule that turns a correlation matrix into loadings. Every method the app offers under factor analysis fits the same common factor model and differs in one thing – how it measures the discrepancy between the observed correlations and the ones the loadings imply – since it is that discrepancy the extraction makes as small as it can, and the named methods are the choices of discrepancy. The choice matters on marginal data – a small sample, few items per factor, a near-singular matrix – and hardly at all on clean structure: at two factors on the café items the nine methods reproduce one another’s loadings within 0.01, within 0.03 for minimum rank, and the comparison dialog’s congruence reads 1.000 for every pair. The factor analysis manual says when to pick which.

Maximum likelihood is the extraction that assumes the items are multivariate normal and finds the loadings under which the observed correlations were the most probable. Its discrepancy is the one that yields a proper χ² test of the model – the chi-square test of model fit – and with it RMSEA, CFI and TLI on the same footing as a CFA’s, since it is the same principle as the CFA’s default estimator. The app prints a χ² for every extraction, computed from this same discrepancy at whatever solution the method reached, so under another method it describes the fit rather than tests it. On the café items it reads χ²(4) = 1.04, p = .90. Its weakness is that it can fail to converge, or end in a Heywood case, on a small sample or a poorly conditioned matrix where a least-squares method still returns a solution.

Principal axis factoring extracts the factors from the reduced correlation matrix by its eigenvectors, as PCA does from the full one, and then iterates: the communalities the loadings imply go back on the diagonal, the extraction runs again, and so on until they stop changing. It assumes nothing about the items’ distribution and needs no likelihood, which made it the traditional choice for ordinal items and small samples, and it is the method most textbooks mean by “factor analysis”. On the café items it matches minimum residual within 0.01 on every loading.

Minimum residual minimises the sum of the squared residuals – each observed correlation minus the one the loadings reproduce – over the off-diagonal of the matrix, directly, with the diagonal free to take whatever communality that implies. It assumes no distribution, converges where maximum likelihood may not, and is what this page’s factor solutions use: on the café items it reads χ²(4) = 1.05, p = .90, RMSEA 0, CFI 1.00.

Unweighted least squares minimises the same squared residuals over the whole matrix, diagonal included. In practice it is minimum residual under another name – the two differ only in whether the diagonal enters the sum – and on the café items they return the same loadings to every decimal, so selecting both in the comparison dialog is not an independent check.

Weighted least squares is the least-squares criterion with every squared residual weighted before it is summed, the weight for a pair of items being the product of their uniquenesses as the inverse of the correlation matrix gives them – 1 minus each item’s squared multiple correlation – so that the residuals among the items the rest of the set predicts poorly count for more than those among the items it predicts well. On the café items it agrees with minimum residual within 0.01.

Generalized least squares weights the residual matrix by the whole inverse of the correlation matrix rather than by its diagonal alone, so that the overlap among the items – not only each item’s own predictability – decides how much each residual counts. It is the least-squares criterion closest to maximum likelihood’s, and on the café items it agrees with minimum residual within 0.01.

Minimum sample size weighted χ² is the least-squares criterion with each squared residual weighted by the number of cases its correlation was computed from – built for a matrix of pairwise correlations whose pairs have different sample sizes because of missing answers, so that a correlation from 50 cases moves the solution less than one from 200. With no missing data every pair has the same count and it is the least-squares fit again: the café items give χ²(4) = 1.04, p = .90 and loadings within 0.01 of minimum residual.

Minimum rank chooses the communalities so that the reduced correlation matrix stays a proper matrix – no negative eigenvalues – and the common variance the retained factors leave unexplained is as small as it can be, so that the share of common variance a factor count explains is a true proportion rather than a figure that can run past 100%. It is the method to reach for when that share is the question; it needs the Rcsdp package, which the app carries. On the café items it differs most from the others: value loads 0.67 against minimum residual’s 0.64, the factors correlate .38 against .39, and the two factors explain 44.9% of the variance against 44.1%.

Alpha factoring treats the items as a sample from a larger set that could have been asked, and the people as fixed, and extracts the factors that generalise best to the items not asked – the ones with the highest Cronbach’s α – by rescaling each item by its communality before the extraction and back after. On well-behaved data its answer is the others’: the café items come within 0.01 of minimum residual. It earns its place when the items differ widely in how much they share.

Congruence coefficient is Tucker’s measure of how alike one factor is in two solutions: the cosine of the angle between the two columns of loadings, 1 when they are proportional and 0 when they have nothing in common. It is read without its sign, since a factor that comes out reflected is the same factor, and it ignores the size of the loadings – which is why a component and a factor built on the same items can read 0.996 although the component’s loadings are uniformly larger. It is a statement about one factor, not a solution: two extractions can agree on Food and disagree on Service, so the app’s comparison table prints a coefficient per factor with the mean beside it and reports the lowest single-factor value separately. The reading is ≥ 0.95 for the same factor, 0.85 to 0.94 for a similar one, below 0.85 for factors that differ (Lorenzo-Seva and ten Berge 2006). Every pair of the café’s nine factor extractions reads 1.000 on both factors, and when the Step 1 method is PCA its components are compared too, at 0.995 to 0.998 against each of them – clean structure, where the method does not matter. A poor congruence looks like a factor that is Food under one extraction and, under another, a taste–value pair with fresh gone over to Service: the two columns point in different directions, and the two solutions are not two estimates of the same thing. A personality researcher meets it when a five-factor solution fitted in a new sample brings back four factors above 0.95 and the fifth at 0.70 – four constructs replicated and one did not.

Factor loading is the correlation between an item and a factor when the factors are uncorrelated, and the item’s regression weight on the factor when they are not: how much of the item is the factor. Loadings run from −1 to 1; taste at 0.76 on Food is a strong indicator, fresh at 0.59 a fair one. Squared, a loading is the share of the item’s variance that the factor explains – 0.76² = 0.58 for taste. A loading below about 0.3 says the item barely belongs, and the app’s table blanks loadings below its display threshold so that the structure is easier to read. A negative loading means the item runs the other way – slow, read as the members answered it, loads −0.61 on Service, as strongly as its neighbours with the opposite sign, which is a sign of the wording and not a flaw.

Loadings matrix is the table of loadings, items as rows and factors as columns – the core output of a factor analysis, and the one to report. Read it row by row to see what each item measures, and column by column to see what each factor is made of.

Factor loadings is the same table under the heading an EFA gives it – the loadings of items on factors.

Component loadings is the table under PCA’s heading – the loadings of items on components, larger than factor loadings on the same items because a component carries the item’s own variance too.

Pattern matrix holds the loadings as regression weights – each item on each factor, the other factors held constant – and is the table to read after an oblique rotation, because a weight of 0.03 for taste on Service means that Service adds nothing to taste once Food is accounted for. It is what the app prints as the loadings under an oblique rotation.

Structure matrix holds the plain correlations between items and factors, which under an oblique rotation are inflated by the factor correlation: taste correlates 0.33 with Service in the structure matrix because Food and Service correlate 0.39 and taste is Food’s. The two matrices coincide when the factors are uncorrelated. Read the pattern matrix to decide what an item measures and the structure matrix to see what it correlates with.

Communality is the share of an item’s variance that the factors explain together – for uncorrelated factors the sum of its squared loadings, and with correlated ones a little more. taste has a communality of 0.60 under the two-factor solution: 60% of the spread in its answers is food satisfaction, 40% is the item’s own. A communality below about 0.3 marks an item the factors do not reach; the six café items run from 0.36 (friendly, slow) to 0.60. Under PCA the same items read 0.56 to 0.69, since a component absorbs the item’s own variance too.

Communalities is the column of them, one per item, and the mean communality – 0.44 here – is the share of the items’ total variance the factors explain, the same number as the cumulative variance explained.

Uniqueness is 1 minus the communality: the share of an item that is not the factors – its own specific content plus its measurement error. taste is 0.40 unique. In a CFA the same quantity is the item’s residual variance.

Eigenvalue is the amount of the items’ total variance that one factor or component accounts for, in units of one item: the six café items have six units of variance, and their eigenvalues are 2.44, 1.31, 0.66, 0.58, 0.54 and 0.47, summing to six. The first component carries 2.44 items’ worth, the second 1.31, and the remaining four less than one item each – which is the whole basis of the Kaiser criterion and the scree plot.

Variance explained is the eigenvalue as a percentage of the total: the two components explain 40.7% and 21.8%, 62.5% together; after extraction and rotation the two factors’ sums of squared loadings, 1.36 and 1.28, are 22.7% and 21.4% of the total, 44.1% together – and, of what the solution explains, 51.6% and 48.4%. The rotated figures are the ones to report, and how much is “enough” depends on what the items are – 40 to 60% is usual for questionnaire items, whose own noise is large.

Rotation turns the factor axes to make the loadings easier to read. The unrotated solution puts as much variance as possible on the first factor, so every item loads on it and the second factor is a contrast; rotating the axes toward the item clusters gives each item one large loading and small ones elsewhere, without changing the communalities or the fit. On the café items the unrotated first factor loads on all six, 0.47 to 0.67 in size (slow at −0.48), and the rotated one on the food trio alone.

Orthogonal rotation keeps the factors uncorrelated, which makes the loadings plain correlations and the results easy to describe, at the price of forcing a zero correlation between constructs that may well be related – a price paid in cross-loadings, since the only way for two orthogonal factors to reproduce related constructs is for their items to load on both: on the café items varimax leaves taste and fresh at 0.18 and 0.28 on Service (mean complexity 1.13), where oblimin, free to correlate the factors, reads 0.03 and 0.16 (1.03). Varimax is the usual choice.

Oblique rotation lets the factors correlate and reports the correlation. Food and Service correlate 0.39 under oblimin – food satisfaction and service satisfaction go together – and an orthogonal rotation would have hidden that. It is the better default for psychological constructs, which are rarely independent, and it costs only the two-matrix reading above. Oblimin is the usual choice.

Rotation criterion is the function of the loadings that a named rotation makes as small or as large as it can – every rotation is defined by one, and the axes are turned until it can improve no further. Each criterion is a different way of writing down simple structure: varimax asks for columns with a few large loadings and many small ones, quartimax for rows with one large loading, geomin for every row to hold at least one near-zero, promax for closeness to a target in which the small loadings have been squashed. On clean structure they agree, as the café’s rotations below do; on data whose items load in more than one way, the criterion decides which pattern the reader is shown, and the factor analysis manual says when to pick which.

Varimax is the standard orthogonal rotation: it maximises the variance of the squared loadings within each factor, so each factor has a few large loadings and many small ones.

Oblimin is the standard oblique rotation, doing the same with the factors free to correlate. It is what this page uses.

Quartimax simplifies the rows: it maximises the sum of the fourth powers of the loadings, which is largest when each item has one large loading and the rest near zero, without caring how the large loadings are shared out among the factors. Its known pull is toward a general factor, since one column that every item loads on makes every row simple. On the café items, with Kaiser normalization on, it is barely distinguishable from varimax – 1.35 and 1.30 items’ worth of variance on the two factors against varimax’s 1.36 and 1.29; with it off the pull shows, 1.43 and 1.22.

Equamax is the compromise between varimax and quartimax, weighting the simplicity of the columns and of the rows so that the variance is spread more evenly across the factors than quartimax allows and a general factor is discouraged more than varimax does. With two factors it comes to the same thing as varimax – on the café items the two agree within 0.01 – and it separates from varimax as the factor count grows.

Varimin is varimax reversed: it minimises the variance of the squared loadings, seeking the rotation on which every item loads as evenly as possible on every factor. It is for the analyst whose theory says the items share a general factor that a simple-structure rotation would hide. On the café items it hands back the unrotated shape – every item 0.46 to 0.66 in size on the first factor (slow negatively) and a ±0.2 to 0.4 contrast on the second, complexity 1.68 – which is what the absence of simple structure looks like.

Geomin minimises, summed over the items, the geometric mean of each item’s squared loadings (with a small constant added so that one zero does not end the search), which is smallest when every item has at least one loading near zero and says nothing against an item having two large ones. That makes it the rotation for scales whose items are expected to load on more than one factor – the usual choice in exploratory structural equation modelling – and the one least likely to hide a real cross-loading. Geomin T is the orthogonal form and Geomin Q the oblique; on the café items the oblique form gives the same trios as oblimin with the factors correlating .38.

Bentler’s invariant is Bentler’s invariant pattern simplicity criterion: it works on the squared loadings and asks that the factors’ patterns of squared loadings overlap as little as they can – be as close to orthogonal as the data allow – judged by a ratio of determinants that does not change when the loadings are rescaled, hence “invariant”. Bentler’s invariant T is the orthogonal form and Bentler’s invariant Q the oblique; in practice the first lands close to varimax and the second close to oblimin, and on the café items the Q form gives the two trios with the factors correlating .41.

Promax takes the varimax solution, raises every loading to a power – the fourth, keeping the sign – so that the small ones shrink toward zero and the large ones keep their standing, and finds the oblique rotation that comes closest to that target. It is fast and needs no iteration, which made it the standard oblique rotation before the oblimin family was cheap to compute, and it tends to let the factors correlate a little more than oblimin does: on the café items φ = .42 against oblimin’s .39, with the same loadings to within 0.02.

Quartimin is the oblique counterpart of quartimax: over every pair of factors it minimises the sum across items of the product of the two squared loadings, so that an item large on one factor is small on the other. It is the member of the oblimin family at γ = 0, and it is what the app’s Oblimin runs, so the two options return the same solution – on the café items to every decimal.

Simplimax goes straight for a chosen number of zeros: it picks the loadings to be made small – by default as many as there are items – and minimises their sum of squares, so the search is for a pattern with that many near-zeros rather than for a property of the whole matrix. On the café items it gives the two trios with the factors correlating .42.

Cluster builds its own target: the varimax solution is turned into a matrix of ones and zeros – every item assigned to the one factor it loads on most – and the loadings are rotated obliquely toward it. It is the most literal reading of simple structure, each item on exactly one factor, and it is at its best where the items really do fall into clean groups: on the café items it gives the two trios with the factors correlating .41.

Kaiser normalization scales each item’s row of loadings to unit length before the rotation and back to its size after, so that the rotation criterion weighs every item equally – an item with a low communality has as much say in where the axes land as one with a high one, instead of the well-measured items settling the axes among themselves. The app’s checkbox follows the rotation’s convention – ticked when varimax is chosen, cleared when any other rotation is – and this page’s oblimin numbers, computed with it on, – taste, fresh and value at 0.76, 0.59 and 0.64, friendly, quick and slow at 0.58, 0.72 and −0.61, φ = .39 – are the normalized rotation; without it the same fit reads 0.77, 0.60 and 0.65, 0.58, 0.72 and −0.60, with φ = .41, a shift of 0.02 at most. It matters more where the communalities are uneven, and quartimax above shows the difference it can make.

Simple structure is the goal rotation aims at: every item loading strongly on one factor and weakly on the rest, every factor having a distinct set of items. The café solution has it – no item loads above 0.17 on its second factor – and the app’s mean complexity of 1.03 says so, since a complexity of 1 is an item on exactly one factor.

Complexity is Hoffman’s index of how many factors an item loads on: 1 for an item on exactly one factor, 2 for one loading equally on two, and in between for the mixtures – computed from the squared loadings as their sum squared over the sum of their squares, so that a small secondary loading counts for little. taste at 0.76 on Food and 0.03 on Service has a complexity of 1.00; under varimax, at 0.75 and 0.18, it reads 1.12, and fresh at 0.61 and 0.28 reads 1.40. The mean complexity over the items is the app’s one-number summary of simple structure: 1.03 for the café’s oblimin solution, 1.13 under varimax, and 1.68 with no rotation at all, where every item loads on the first factor and most on the second too – the difference rotation makes, in one number. It is reported for every candidate count in the model comparison table and for the chosen one under Solution quality, and a mean above about 1.2 says the solution is not simple: either the rotation is wrong for the data, or the items really do measure two things at once.

Cross-loading is an item that loads substantially on two factors: fresh at 0.59 on Food and 0.16 on Service is the closest the café items come, and 0.16 is not substantial. A true cross-loading, both above about 0.3, is an item that asks two things at once and is usually dropped or rewritten.

Salient loading is a loading large enough to count – conventionally 0.3 in absolute value, the size at which a factor explains about a tenth of an item’s variance – and the bar an item must clear on at least one factor to belong to the solution. The count of items that clear it on none is the app’s Unassigned variables in the model comparison table and Variables with no salient loading under Solution quality: 0 for the café items at every count, since even the weakest, friendly and slow, load 0.58 and −0.61 on Service – the bar is on the size, whichever the sign. An item with no salient loading is measuring something the factors do not cover, or nothing at all, and is the first candidate to drop; a solution with several is one factor short, or is being asked to explain items that were never a scale. The 0.3 is fixed and inclusive, and it does not move with the display threshold that blanks small loadings in the table; a loading of 0.3 is salient by this rule and still a weak indicator by the loading block’s reading, which is the difference between belonging to a factor and measuring it well.

Factor correlation is the correlation between two factors under an oblique rotation, or between two latent variables in a CFA: 0.39 for Food and Service here, 0.45 in the CFA below. It is larger than the correlation between the two mean scores built from the same items (slow reversed in Service’s, as a scale score reverses it), which is 0.30, because the mean scores carry measurement error that pulls their correlation towards zero and the latent variables do not. That gap – observed 0.30, latent 0.45 – is what “correcting for attenuation” means, and it is one of the main reasons to model latent variables rather than sums.

Factor correlations is the matrix of them, one factor correlation per pair; under an orthogonal rotation or PCA it is an identity matrix and says nothing.

Factor scores are estimates of each person’s standing on each factor – a member’s food satisfaction as a number – computed from their item answers with weights derived from the model. They are estimates, not observations: the model does not fix them, so a scoring method has to choose them, and the app’s determinacy index – 0.86 for Food and 0.84 for Service under the default regression method – is the correlation between the score and the factor it estimates. Two scores built by different methods from the same solution disagree, and their correlations with one another need not match the factors’ – the café’s regression scores correlate 0.51 where the factors correlate 0.39, which is the same estimation error read from the other side. Insert them into the dataset when a later analysis needs a person-level number, and prefer a plain mean of the items when the loadings are similar – it is simpler to report and nearly as good.

Factor score estimation is the step from a fitted solution to a number per person, and it is a step with a choice in it, because the common factor model does not determine the scores: a factor is defined by what its items share, and infinitely many sets of scores are consistent with the same loadings, communalities and factor correlations (Grice 2001). Every scoring method is a rule for choosing one set among them – each computes a person’s score as a weighted sum of their standardized items, and the methods differ in the weights – and each rule buys one property at the cost of another: regression scores the highest correlation with the factor, Bartlett scores freedom from bias, Anderson–Rubin scores zero correlation among the scores, Ten Berge scores the factor correlations reproduced exactly. On the café’s two-factor oblimin solution the four rules give Food and Service scores that correlate 0.51, 0.26, 0.00 and 0.39 against a factor correlation of 0.39 – one solution, four answers to a question the model leaves open. With one factor there is nothing to weigh against and the four rules agree, so the app fixes the choice to regression there. A component has none of this: it is an exact weighted sum of the items, its scores are computed rather than estimated, and under PCA the choice only decides which property the scores are made to carry. A teacher saving an ability score per pupil or a manager an engagement score per employee makes the same choice, whether or not the software says so.

Regression scores are the estimates that correlate most highly with the factor they stand for – Thurstone’s method, the app’s default and what most software means by “factor scores” (Thurstone 1935). The weights are the regression weights of the factor on the items, computed from the item–factor correlations of the structure matrix and the correlation matrix, since the factor itself is never observed to regress on. The price of the fit is bias: the scores are pulled towards the mean, so a member with an extreme true standing gets a less extreme estimate, and the scores correlate with one another as the items dictate rather than as the model does – on the café items the regression scores for Food and Service correlate 0.51 against a factor correlation of 0.39, and under a varimax rotation, where the factors are uncorrelated by construction, they still correlate 0.15. Choose them when the scores are wanted for what they are – ranking or profiling people – and something else when their correlations or their spread will be read as the factors’.

Bartlett scores are the unbiased estimates: the weights are chosen so that a person’s expected score equals their true standing on the factor, with no pull towards the mean, by weighting each item by the inverse of its uniqueness – the items the factor explains best count most – and making the unique parts of the items contribute as little as possible to the score (Bartlett 1937). What is given up is a little determinacy, 0.86 and 0.83 on the café items against regression’s 0.86 and 0.84, and any promise about the correlations among the scores, which can fall short of the factors’ as easily as overshoot them: the café’s Bartlett scores correlate 0.26 against a factor correlation of 0.39. The usual advice is to prefer them when the scores will be an outcome or compared between groups, where a shrinkage of every score towards the mean matters more than a little extra noise.

Anderson–Rubin scores are Bartlett’s method with a constraint added: the scores come out uncorrelated with one another, each with a mean of 0 and a variance of 1 (Anderson and Rubin 1956). That is a convenience when the scores are to be predictors in a regression – uncorrelated predictors have no collinearity – and a contradiction whenever the rotation was oblique: the model says the factors correlate, the scores say they do not. On the café items the Anderson–Rubin scores for Food and Service correlate 0.00 against a factor correlation of 0.39, and the constraint costs determinacy, 0.84 and 0.80 against regression’s 0.86 and 0.84, the lowest of the four methods. The app allows the combination and warns under an oblique rotation; a correlation among these scores is a property of the method, never an estimate of the factor correlation. The same happens under PCA when the components correlate under an oblique rotation – 0.27 on the café items, where the Anderson–Rubin scores read 0.00.

Ten Berge scores are the correlation-preserving estimates: among the sets of scores consistent with the solution, the one whose correlations equal the factor correlations of the model, chosen to correlate with the factors as highly as that constraint allows (Ten Berge, Krijnen, Wansbeek and Shapiro 1999). On the café items they correlate 0.39 between Food and Service, the factor correlation to the decimal, at a determinacy of 0.86 and 0.84 – regression’s to two decimals – so nothing is given up for the property here. They are the method to choose when the scores will stand in for the factors in an analysis that reads their correlations – a regression among them, a comparison of profiles – since regression scores overstate the factor correlation and Anderson–Rubin scores erase it. Their one weakness is numerical: the weights need a matrix built from the loadings and the correlation matrix inverted, and when it cannot be the app falls back to regression weights and says so.

Score determinacy is the correlation between the estimated scores and the true factor they estimate – one number per factor, for the scoring method actually used – reported as ρ rather than as its square, because that is the scale the rules of thumb are written on: ρ ≥ 0.80 is the accepted minimum and ρ ≥ 0.90 the preferred bar (Gorsuch 1983; Grice 2001), and “factor determinacy” in other software is the same number. It is computed from the chosen method’s weights, so it grades the scores in the table rather than the solution in the abstract, and switching methods changes it: on the café’s oblimin solution regression scores read 0.86 for Food and 0.84 for Service, Ten Berge scores the same, Bartlett scores 0.86 and 0.83 and Anderson–Rubin scores 0.84 and 0.80 – every method above the minimum, none at the preferred bar, which is what three items per factor with communalities of 0.36 to 0.60 buy. A low value means two members with very different true standings can land on similar scores, and anything built on the saved column inherits that blur; more items, or better ones, raise it, and no scoring method can raise it far. Under PCA regression scores read 1.000, because a component is an exact weighted sum of the items and regression weights recover it perfectly – so do Ten Berge scores – while Bartlett and Anderson–Rubin scores land just below, 0.999 and 0.99 on the café items, since each imposes a property the component does not have.

Score coefficients are the weights the scores are built from: one per item per factor, applied to the standardized items – multiply each item by its coefficient in a factor’s column, add the products, and the sum is that person’s score on the factor. They are the recipe a score column can be recomputed from, or applied to a fresh sample without refitting, which makes the coefficient matrix the reproducibility record of a saved score. They are not loadings and need not resemble them: a loading runs from the factor to the item and says how much of the item the factor explains; a coefficient runs the other way, from the items to the score, and depends on the scoring method and on how much the items overlap – an item whose ground another item already covers adds little to the score however well the factor explains it. On the café’s Food factor under regression scoring taste has a coefficient of 0.49 and fresh 0.28 where their loadings are 0.76 and 0.59: fresh loads three-quarters as strongly and weighs little more than half as much, because most of what it says about Food taste has already said. The service items weigh no more than 0.04 either way on Food, near nothing, and Bartlett scoring weights the food items 0.70, 0.38 and 0.37 instead – a different recipe for the same factor. Report loadings to describe the structure and coefficients to say how the scores were made.

Score correlations are the correlations among the saved score columns, computed from the scores as the method produced them – a property of the estimates rather than of the factors, and the check on the method. They are read against the model’s factor correlations: Ten Berge scores reproduce them exactly, Anderson–Rubin scores read 0 whatever the model says, and regression and Bartlett scores do neither – they track the factor correlations roughly under an oblique rotation and show some correlation under an orthogonal one, where the factors have none. On the café’s oblimin solution, with Food and Service correlating 0.39, the score correlations run 0.51 under regression, 0.26 under Bartlett, 0.00 under Anderson–Rubin and 0.39 under Ten Berge, and under varimax the regression scores still correlate 0.15. The gap between a score correlation and the factor correlation is indeterminacy read from the other side, and it is why a correlation computed later between two saved columns is not an estimate of the correlation between the constructs – the model’s factor correlation is. Under PCA regression and Ten Berge scores reproduce the component correlations exactly and Anderson–Rubin scores read 0 against them – 0.27 on the café items – with Bartlett scores close but not bound to match.

Heywood case is a solution with an impossible value – a communality above 1, or a negative residual variance – which means the model is wrong for the data or the sample too small for it: too many factors extracted, an item that is a near-copy of another, or fewer than a handful of items per factor. An extraction that keeps every residual variance above a small floor never shows the impossible value: the communality stops just short of 1 and the residual variance sits on the floor, which is the same case reached from the other side. The app flags it either way; the fix is a different number of factors or a different set of items, not a different rotation.

Can the items be factored? #

A factor analysis is fitted to a correlation matrix, and not every matrix can be factored. Before a count is chosen the app checks four things: that the items share enough variance for common factors to exist, that the matrix is internally consistent, that no item is a copy or a sum of the others, and that enough cases stand behind the correlations. The six café items pass all four, so the numbers in this section are what a healthy matrix looks like, and the failures are described in words.

KMO – the Kaiser–Meyer–Olkin measure of sampling adequacy – says whether the items share enough variance for a factor analysis to be worth running: the share of the correlations among the items that is not explained away by the partial correlations among them. It runs from 0 to 1 and 0.6 is the usual floor. The café items give 0.73 overall and 0.70 to 0.76 per item – adequate, and typical of a short scale. A low per-item KMO marks an item that correlates with nothing else and belongs to no factor.

Bartlett’s test of sphericity tests whether the correlation matrix is an identity matrix – all items uncorrelated – in which case there is nothing to factor. It rejects on the café items, χ²(15) = 229.1, p < 0.001, and it rejects on nearly every real dataset of any size, so a pass is a necessary condition and never evidence that the analysis is sound. The measure with information in it is KMO.

Anti-image correlation is the negative of the partial correlation between two items with all the others held constant. In a set that factors well most of them are small, since what two items share is shared through the factor; a large one marks two items that share something private to them – taste and fresh at −0.36 are the café’s largest. The diagonal holds each item’s own KMO.

Anti-image covariance is the covariance form of the anti-image correlation – the same matrix before it is standardized. Its diagonal is each item’s anti-image variance, the share of the item the other items cannot predict, 1 minus its squared multiple correlation: a small value marks an item that shares most of itself with the set, which is what makes it worth factoring, and the café items run from 0.64 (taste, the best predicted) to 0.77 (slow). The off-diagonals are what survives of each pair’s association once every other item is partialled out, and should sit near zero; taste and fresh at −0.24 are the café’s largest, the same pair the correlation form puts at −0.36. Dividing each cell by the square roots of its two diagonal entries gives the correlation form, whose diagonal the app then replaces with each item’s KMO; both need the correlation matrix inverted, so both are withheld on a singular one.

Positive definite is the property a correlation matrix must have for its correlations to be mutually consistent: every eigenvalue above zero. Three items of which the first correlates 0.9 with each of the other two cannot have those two correlating −0.9 with each other – no set of numbers behaves that way – and a matrix that claims they do has a negative eigenvalue. The same happens, less visibly, when a matrix is assembled pair by pair from different subsets of cases, or by an estimator that treats each pair on its own, as polychoric correlations do. The check is the smallest eigenvalue: the café items’ is 0.47, comfortably positive, and the app’s suitability card reports it as Matrix is positive definite, graded against a small tolerance rather than a bare zero so that a matrix a hair from singular counts as failed. A matrix that fails is smoothed before extraction by the methods that can work on the result, and refused by maximum likelihood, alpha factoring and minimum rank, which cannot.

Smoothing is the repair applied to a correlation matrix that is not positive definite: the eigenvalues at or below zero are lifted to a small positive value and the matrix is rebuilt and rescaled so that it is a correlation matrix again, with 1s on its diagonal. The extraction then runs on the smoothed matrix, and the price is that the correlations it fits are no longer quite the ones observed – close, since only the impossible part is changed, but the loadings, the fit indices and every card downstream describe the repaired matrix, and the app says so above each of them. Smoothing repairs the numbers and not their cause: an item that made the matrix singular is as redundant afterwards as before, and the rank check is what names it. The app smooths for every extraction that can use the result; maximum likelihood, alpha factoring and minimum rank stop instead. The café matrix needs none.

Singular matrix is a correlation matrix in which one variable is exactly reproducible from a combination of the others – a total score analysed beside the items it sums, a difference beside the two numbers it was taken from, an item entered twice. It has a determinant of zero, an eigenvalue of zero and no inverse, and it need not look wrong: a total of five items correlates only 0.5 to 0.7 with each of them, so every pairwise correlation stays modest while the matrix as a whole cannot be factored. That is why the app checks the rank directly rather than trusting the pairwise correlations, and why KMO and Bartlett’s test, both of which need the inverse, are marked Unreliable (singular matrix) on one, with the anti-image partials withheld for the same reason. The café items are six answers to six different questions and their matrix is not singular; a teacher who factors an exam’s section scores together with its total, or a manager who includes an engagement index beside the items it averages, has built one.

Matrix rank is the number of variables in a correlation matrix that carry information of their own – the columns that cannot be rebuilt from the others. A healthy matrix has full rank, one per variable; the café’s six items have rank 6. A rank below the number of variables is a singular matrix, and the shortfall says how many variables are redundant: five items and their total have rank 5. The app computes the rank on its suitability card, and when it falls short names each redundant variable with the variables it is reproduced from, written as Total ← Item A, Item B, Item C, so the fix – drop the total, keep its parts – is read off the alert. It is the condition that multicollinearity names among a regression’s predictors, taken to its limit, and it has the same cure.

Correlation matrix determinant is one number that summarises how far a correlation matrix is from singular – the product of its eigenvalues: 1 when every correlation is zero, 0 when the matrix is singular, and smaller the more the items overlap. The café’s six items give 0.31, well clear of zero. Read it with the number of items in mind, though: the determinant shrinks with every variable added whatever the data look like – items correlating 0.3 throughout give 0.42 when there are six of them, 0.0003 at thirty and 0.00001 at forty – so a fixed bar such as Field’s 0.00001 is crossed by every forty-item questionnaire, redundant or not. The app reports the number and the bar and withholds the Multicollinearity detected verdict unless a check that does not depend on the item count agrees – the rank, a pair correlating above 0.9, or a squared multiple correlation above 0.99.

Cases-to-variables ratio is the number of complete cases per item – the café’s 200 members over six items are 33 to 1 – and the app’s summary of whether the sample is large enough for the correlations to be trusted. The rules of thumb are many and disagree: 50 cases as an absolute floor, 5 or 10 per item, 300 for a stable solution. The app grades the ratio against the lowest of them – Sample too small below 50 cases, Below the recommended 5:1 under five per item – because what decides how many cases a factor analysis needs is the strength of the structure rather than any ratio: items with communalities around 0.7 recover their factors in a hundred cases, and items around 0.3 need several hundred to come out the same in a second sample (MacCallum, Widaman, Zhang and Hong 1999). A high ratio does not make a weak structure stable and a modest one does not spoil a strong one; the KMO and the size of the loadings are the better guide.

How many factors #

The number of factors is the analyst’s decision, and the app offers several rules that often disagree. On the café items the eigenvalue-greater-than-one rule and its empirical correction, parallel analysis under both of its nulls, the Hull method and VSS₁ all say two; Velicer’s MAP says one, and the elbow and VSS₂ say three. The disagreement is not a malfunction. The second factor is real – the two clusters are visible in the correlations – but small, and a rule built to guard against over-extraction stops at one, while the elbow, which reads the shape of the curve alone, tends to run one high, and VSS₂ on six items has only two counts to choose between. The survey was written with two scales in mind, and the two-factor solution fits where the one-factor one does not (model comparison), so two it is; when the rules disagree, theory and fit decide.

Scree plot plots the eigenvalues in order, largest first. The curve drops steeply while each factor is picking up real structure and flattens once the remaining factors are noise; the number of factors is read at the bend. On the café items the line falls from 2.44 to 1.31 to 0.66 and then creeps down – the bend is at the third.

Elbow is that bend, and the point of judgement. The app locates it as the point on the curve furthest from a straight line drawn from the first eigenvalue to the last, marks it with an enlarged dot, and reports that point as the count – the third eigenvalue on the café items, one more than the Kaiser rule and parallel analysis give. That is typical: the corner of a curve that drops and then flattens is usually the first flat point, so the elbow count runs one high on clean structure and is a check on the other rules rather than a rule of its own.

Kaiser criterion keeps every component whose eigenvalue exceeds 1 – one item’s worth of variance – on the reasoning that a factor explaining less than one item is not worth having. It gives two on the café items (2.44 and 1.31 above, 0.66 below) and it is known to over-extract on large item sets, since in any finite sample the first eigenvalues of pure noise are above 1 too. It is read on the correlation matrix’s own eigenvalues and never on the reduced series, whose average is below 1, and the app draws its bar as a dashed line at 1 on the scree plot; the empirical Kaiser criterion is the same rule with the bar corrected for the sample’s size.

Empirical Kaiser criterion is Kaiser’s rule with the bar of 1 replaced by a reference eigenvalue for each position, derived from what random data with the sample’s number of cases and items would produce (Braeken and van Assen 2017). Pure noise does not give eigenvalues of exactly 1: in a finite sample the largest is above 1 by an amount that grows as the cases per item shrink, so a flat bar keeps noise factors, and the reference corrects for that. The first reference is the ceiling for noise at the sample’s size – 1.38 for 200 cases and six items – and each later one is that ceiling rescaled by the variance the earlier factors have already claimed, never below Kaiser’s 1; on the café items the second and every later reference is exactly 1.00. Factors are kept up to the first that fails its reference: 2.44 beats 1.38 and 1.31 beats 1.00, 0.66 does not, so two – the same count as Kaiser here, since the café’s second eigenvalue is well clear of either bar, and fewer than Kaiser on a large item set with little structure, which is where it is the count to prefer. It costs no simulation, it is read on the same series as Kaiser, and the app draws its reference as a brown dashed line on the scree plot.

Parallel analysis compares each eigenvalue with the eigenvalues that random data of the same size – as many cases, as many items, and no structure at all – would produce, and keeps a factor only while the real eigenvalue is larger than the 95th percentile of the random ones (Horn 1965). The classic null simulates normal data afresh for each of a hundred draws (the app’s default), and it is what the app reports as Parallel analysis (simulated data): on the café items the reference points are 1.33, 1.18 and 1.08 for the first three positions; the real eigenvalues 2.44 and 1.31 beat theirs and 0.66 does not, so two. It is the rule to prefer when the others disagree, because it asks the right question – is this factor more than chance would give – rather than applying a fixed cut-off, and the app runs it twice, against this null and against a resampled one.

Resampled parallel analysis is the same rule with a different null: instead of simulating normal data, the app shuffles each item’s own column of answers independently of the others, which destroys every correlation and keeps every item’s distribution – its skew, its ties, the floor or ceiling a 1–5 answer runs into – exactly as observed (Buja and Eyuboglu 1992). The two nulls ask the same question of different pretend data, and they part where the items are skewed or coarse: a five-point item on which most answers are 4 or 5 produces chance correlations of a size normal data never would, so the resampled reference sits higher and keeps fewer factors, and it is the one to prefer for ordinal and skewed items. On the café items, symmetric answers on a 1–5 scale, the two nulls coincide: at the 95th percentile the simulated reference runs 1.33, 1.18, 1.08 and the resampled 1.31, 1.19, 1.09, both say two, and the app lists the two counts side by side as Parallel analysis (simulated data) and Parallel analysis (resampled data).

Two eigenvalue series is what an exploratory factor analysis has where a PCA has one. The correlation matrix’s eigenvalues – 2.44, 1.31, 0.66, 0.58, 0.54, 0.47 on the café items, summing to six – describe the items’ total variance, and the reduced correlation matrix’s – 1.73, 0.58, −0.09, −0.15, −0.19, −0.20 – describe the common variance the extraction works on, with a negative tail that is routine. Each rule reads the series it was defined on. The Kaiser and empirical Kaiser criteria are bars set against the average eigenvalue of a matrix with 1s on its diagonal, so they read the first series – on the reduced one Kaiser’s bar would keep a single factor. Parallel analysis reads whichever series the extraction reads, against a null reduced the same way: under a factor method the reduced 1.73, 0.58, −0.09 are compared with reduced references of 0.36, 0.21, 0.10, and two it is on either series. The elbow reads the curve as drawn, tail included. The app’s scree plot draws the reduced series in blue under a factor method with the correlation matrix’s in grey behind it, and its model comparison table carries both as Eigenvalue and λ (correlation matrix); under PCA there is one series and one curve.

Hull method is a retention rule that reads fitted solutions rather than eigenvalues (Lorenzo-Seva, Timmerman and Kiers 2011). Every factor count the model can identify is fitted, from none up to the bound, and each solution is placed by two numbers – how much of the correlations it reproduces, and how many degrees of freedom it spent to do so. Solutions that add no fit over a simpler one are dropped, leaving a frontier – the hull – that climbs steeply while each factor buys real fit and flattens once it does not, and the count at the sharpest turn of that frontier is the recommendation: the last factor that returned more fit than it cost. On the café items it is two, where one factor reproduces the correlations poorly and three, the saturated count, reproduces them perfectly by construction and buys nothing worth its parameters. It is the scree plot’s elbow applied to fit rather than to eigenvalue size, and located by arithmetic rather than by eye; because it weighs parsimony, it can stop below parallel analysis when a second factor is real but small. It is defined for common factors only – the app does not run it under PCA – and left out when the frontier has no turning point to find.

Velicer’s MAP – the minimum average partial – asks what is left among the items once k components have been taken out. After partialling the first k components out of every item, the partial correlations that remain are squared and averaged; while the components being removed are real, each one strips common variance away and the average falls, and once they are noise, each one removes a piece of some item’s own variance instead and the average climbs. The count at the minimum is the recommendation. On the café items the average reads 0.081 after one component, 0.102 after two and 0.211 after three, so MAP says one: the second factor is real but small, and a rule built to stop before noise stops before it. It reads what is left rather than the eigenvalues, which makes it an independent second opinion on parallel analysis, and its known lean is towards too few factors where Kaiser’s is towards too many; the app evaluates it over every count the model can identify and bolds the minimum in its model comparison table.

Very simple structure is a retention rule that asks how well a solution would reproduce the correlations if each item were allowed only its largest loading – or its two largest – and every other loading were set to zero (Revelle and Rocklin 1979). That simplified solution is what a reader of a loadings table actually uses, and a count is right for it when adding a factor still improves what the simplified loadings reproduce. VSS₁ keeps one loading per item and VSS₂ two, each the share of the correlations the simplified solution reproduces, from 0 to 1, and the count at the maximum is the recommendation. On the café items VSS₁ reads 0.61, 0.68 and 0.64 for one, two and three factors, so two; VSS₂, undefined at one factor, reads 0.78 at two and 0.80 at three – the saturated count, where any solution reproduces everything – and a criterion that can only be evaluated at two counts has no room to peak, so on six items the VSS₂ count is not a stable read. It is on the longer scales the rule was written for, where VSS₂ usually peaks where VSS₁ does or one above it. The app computes both under a varimax rotation, since simple structure is what the criterion measures, and bolds each column’s maximum in the model comparison table.

One general factor and several specific ones #

Correlated factors raise one more question. Food and Service correlate 0.39, and a correlation between two factors can be read as a factor above them – a satisfaction with the café as a whole that runs through all six items, with food and service as what each trio measures beyond it. The two models of that structure are one claim written from opposite ends, the Schmid–Leiman transformation is the arithmetic that joins them, and the coefficients that say what the general factor is worth – omega hierarchical and the subscale omegas – are on the reliability page. The numbers below are the café’s two-factor oblimin solution with Schmid-Leiman transformation ticked in the factor module.

Bifactor model is a model in which every item loads on one general factor, shared by all the items, and on one group factor, shared by its own cluster alone – the group factors uncorrelated with the general factor and with one another – so that each item’s variance is split three ways: what it shares with everything, what it shares with its cluster, and what is its own. It is the structure a scale with subscales is usually written to have – an ability test whose verbal, numerical and spatial sections all draw on a general ability; a satisfaction survey whose food and service items all draw on satisfaction with the place – and it is what the factor module’s Bifactor rotation fits in one step, with Biquartimin (oblique bifactor) as the form that lets the group factors correlate; the IRT module offers the same pair of rotations. Both need one factor for the general dimension and at least two for the groups, and on the café’s six items three factors is the Ledermann bound: the one-step fit is saturated, and it comes out with no general column at all – one column on the food trio, one on the service trio and one on value alone (0.77), so the app leaves them numbered and says no general factor could be identified – and with value at a communality of 0.995, its residual variance on the extraction’s floor – a Heywood case. The same fit with slow reversed rotates to a general column instead: the criterion has more than one minimum here, and which one the rotation lands in depends on where it starts – a hierarchy asked of fewer items than can pin it down. The Schmid–Leiman transformation reaches the same structure from the two-factor solution instead, and is the café’s route to it.

General factor is the factor every item loads on – the column a bifactor or Schmid–Leiman solution labels g – and it is what a total score across all the items measures, since a sum counts what the items share with everything in full and each cluster’s specific part only in proportion. Its loadings say how much of each item is the general dimension: in the café’s transformed solution g loads 0.34 to 0.50 in size on the six items – slow negatively, at −0.38 – taste and fresh highest and value lowest, and the general row of the variance decomposition, 1.09 of 2.65 units of common variance, is its share – the explained common variance of 0.41. What the general factor is worth is read on the reliability page: omega hierarchical, 0.44 here, is the share of a total score that is g, and coefficient H, 0.58 for the café’s g against 0.53 and 0.50 for the group factors, is how well an optimally weighted composite of the items would recover it – none of the three well defined by six items, which is what a short survey buys. In a direct bifactor rotation the app identifies the general factor as the column that loads broadly on the items with one sign, and labels it g; when no column does, the columns stay numbered and there is no g to read.

Group factor is a factor for one cluster of items – the food trio, the service trio – carrying what the cluster shares beyond the general factor, and in a bifactor or Schmid–Leiman solution it is orthogonal to g and to the other group factors, so its loadings are the cluster’s specific part alone; the app numbers them F1, F2 … beside g. In the café’s transformed solution the food group factor loads 0.60, 0.46 and 0.50 on taste, fresh and value and the service one 0.45, 0.55 and −0.46 on friendly, quick and slow, with nothing above 0.11 off its own trio: each trio keeps a specific part about the size of its general part, which is what makes Food and Service worth scoring on their own, and the pair of subscale omegas says how reliably. A group factor whose loadings all fall below the display cutoff – an empty column in the hierarchical loadings table – is one the general factor absorbed: the cluster had nothing specific of its own, and the remedy is usually one factor fewer, not a lower cutoff. A general factor with no group factor left beside it is a unidimensional scale, and the two indices below say how close a scale comes to that.

Hierarchical factor model is the same structure written from the other end: the items load on first-order factors alone – Food and Service – and those factors are correlated because a second-order factor above them explains their correlation. It is the reading an oblique rotation invites, since the 0.39 between Food and Service is, under this model, the product of their two loadings on the factor above. A second-order model is a bifactor model with a constraint added – every item’s general loading is its first-order loading times its factor’s second-order loading, a proportionality the direct bifactor rotation does not impose (Yung, Thissen and McLeod 1999) – so the two are one hypothesis at two levels of freedom, and the Schmid–Leiman transformation is the arithmetic that takes the second-order form to the direct one. What a hierarchy needs that a plain oblique solution does not is enough first-order factors: two give one correlation from which two second-order loadings must be estimated, so with two groups the hierarchy is not identified on its own, and three is where it becomes a model the data can contradict. A three-scale wellbeing inventory – sleep, mood and energy items, with general wellbeing above them – is the smallest fully identified case; the café’s two scales show the reading and not the full model.

Schmid–Leiman transformation is the arithmetic that turns a second-order solution into direct general and group loadings (Schmid and Leiman 1957): an oblique factor analysis of the items first, then a one-factor analysis of the factors’ own correlation matrix, and each item’s general loading is its first-order loading times its factor’s second-order loading, its group loading the first-order loading times the square root of what that leaves – the factor’s part the general factor does not explain. In the app it is the Schmid-Leiman transformation option under an oblique EFA rotation, and it is a separate fit rather than a rearrangement of the loadings already shown: psych’s omega() re-extracts and re-rotates the solution before it transforms it, which is why the hierarchical loadings can differ a little from the primary ones and why the card carries warnings of its own about normalization, a substituted rotation or convergence – and about any item omega() reverse-scored before computing the coefficients: slow here, whose loadings the card still shows in its own direction. On the café’s two-factor oblimin solution each first-order factor loads 0.64 on the general factor – the square root of their correlation, .41 in the unnormalized rotation the transformation runs – so taste’s 0.77 on Food becomes 0.50 on g and 0.60 on the food group factor, and over the items g reads 0.34 to 0.50 and the group factors 0.45 to 0.60 on their trios, in size. The general and group rows of the variance decomposition, 1.09 and 0.81 + 0.74, sum to 2.65, the sum of the communalities, because no item cross-loads; with cross-loadings the two totals drift apart a little. Two group factors is the identification edge: with one factor correlation to go on, omega() sets the two second-order loadings equal and the app says so, so the café shows what the transformation does and a hierarchy that could be tested needs three group factors. Schmid-Leiman factor scores are the factor scores of the transformed solution, a g column and one per group factor; because each item’s general and group loadings are proportional by construction they are the least determinate scores the module produces – 0.67 for g and 0.69 and 0.66 for the groups on the café items, below the 0.80 the rule of thumb asks.

Explained common variance is the general factor’s share of the common variance – the sum of its squared loadings over the sum of every squared loading, general and group together – and the one-number answer to how much of what the items share is one thing. It is read over the common variance alone: the café’s six items have 2.65 units of it rather than six, so the decomposition says nothing about uniqueness, and the general row’s 1.09 is an ECV of 0.41, with the food and service group factors at 0.31 and 0.28 – each group factor’s share is its own ECV, and the three sum to 1. A high ECV says a total score is defensible and the subscales add little to it; a low one, as here, says the subscales carry information of their own, which for a survey written to measure food and service separately is what its design expected. The app grades it in three bands – substantial multidimensionality below 0.50, moderate multidimensionality to 0.70, essentially unidimensional from 0.70 – but withholds the last unless PUC is at least 0.80 or omega hierarchical at least 0.70 agrees, since an ECV computed on a matrix with few uncontaminated pairs overstates the case (Rodriguez, Reise and Haviland 2016); its 0.70 is stricter than the 0.60 the published rule of thumb uses.

Percent uncontaminated correlations is the share of item pairs whose two items belong to different group factors – the pairs whose correlation carries the general factor alone, since two items of one cluster correlate through their group factor as well and their correlation cannot tell the two apart. Six items make fifteen pairs; three lie inside the food trio and three inside the service trio, and the nine that cross are 60% – the café’s PUC of 0.60, each item assigned to the group factor it loads on most, which the app does from the data rather than from a subscale key. It is read beside ECV: the general factor’s share is estimated too high when few pairs are in a position to reveal a group factor, so a high ECV on a low PUC overstates unidimensionality, and only from PUC 0.80 can ECV stand on its own (Rodriguez, Reise and Haviland 2016) – below that the app asks omega hierarchical to agree before it calls a scale unidimensional. PUC rises with the number of groups rather than their length: two groups never exceed two thirds and fall towards a half as the scales lengthen, three scales of three items give 0.75, ten of three 0.93 – which is why a many-subscale inventory can be judged on ECV and two three-item scales cannot.

Confirming a structure: CFA and SEM #

Exploration asks the data what the structure is; confirmation states a structure and asks the data whether it holds. The step is worth taking whenever a scale has a documented structure, and it is what makes fit indices possible: a model that could have failed can be said to fit.

Confirmatory factor analysis fits a measurement model the analyst writes down – which items load on which factor, and nothing else – and reports how well the model reproduces the correlations, alongside the loadings. The café model is two lines, Food =~ taste + fresh + value and Service =~ friendly + quick + slow, with the two factors free to correlate. It fits: χ²(8) = 6.41, p = 0.60, CFI 1.00, TLI 1.01, RMSEA 0.000 (90% CI 0.000 to 0.071), SRMR 0.034, and the standardized loadings are 0.77, 0.68 and 0.57 for the food items and 0.62, 0.70 and −0.59 for the service items, all with p < 0.001 – slow negative, since the model reads it as answered. The rival model with one factor behind all six items does not fit at all – the chi-square difference below – and that contrast is what a CFA is for. The CFA chapter of the SEM manual covers the model editor, the options and the results tables.

Structural equation modeling joins a measurement model to regressions among the latent variables it measures, so a hypothesis about constructs is tested with the measurement error taken out. Adding one line to the café’s CFA – overall ~ Food + Service – regresses the members’ 0–10 overall rating on the two latent satisfactions rather than on two mean scores: Food’s standardized effect is 0.49 and Service’s 0.55, both p < 0.001, and together they explain 79% of the variance in overall – on the 193 members who answered it, since the model uses complete cases unless FIML is on. The whole model still fits, χ²(12) = 9.90, p = 0.62. The same regression on the two item means would give smaller coefficients and a lower R², attenuated by the means’ measurement error. The SEM manual has the syntax and the multi-group editor.

Path analysis is a model of regressions among observed variables only – no latent variables, no =~ lines – drawn as a diagram of arrows, so that a chain such as wait → overall → recommend is one model and its indirect effect can be estimated and tested. DataSuite offers it twice: the regression module’s Advanced tab fits each equation on its own and bootstraps the products along the chains, which is exact for a recursive model and is where a test of directed separation asks whether the arrows left out are justified; the SEM module estimates the same equations jointly, with a full set of fit indices, and is the tool when a latent variable enters or a rigorous categorical path model is wanted.

Test of directed separation is the fit test of a path model estimated equation by equation, due to Shipley: every pair of variables the diagram joins by no arrow is a claim that the two are independent once the parents of both are held fixed, each claim is tested by adding one variable to a regression of the other on that conditioning set, and Fisher’s $C=-2\sum \ln p$ combines the claim p-values into one statistic, χ² on twice the number of claims when every claim holds. Drawing overall ~ taste + wait and recommend ~ overall on the 178 café members complete on all four leaves two claims – that taste and wait each tell nothing about recommend once overall is known – tested by adding each to the logistic regression of recommend on overall: likelihood-ratio χ²(1) = 2.50, p = .114 for taste and 0.03, p = .862 for wait, so C = 4.64 on 4 df, p = .326, and the two arrows the model leaves out are consistent with the data. A significant C says the opposite – that an arrow not drawn belongs in the model – and the claim carrying most of the statistic names which one; it is the one check that can contradict a path model as a whole, since each equation’s own output never can.

Structural model is the regression part of an SEM – the ~ lines among latent and observed variables – as opposed to its measurement model. Its coefficients are read like any regression coefficient: a one-unit rise in Food raises overall by 1.05 points; in standardized form, one SD of Food is 0.49 SD of overall.

Parameter estimates are the numbers a fitted model consists of – every loading, regression, variance and covariance – each with its standard error, z, p and confidence interval, in unstandardized and standardized form. The tables of them are the model’s results; the fit indices are the summary of how well the model as a whole reproduces the data.

Marker variable is the indicator whose loading is fixed to 1 to give a latent variable a scale, since a variable that is not observed has no units of its own. In the café CFA taste is Food’s marker and friendly Service’s – the first item of each – so the unstandardized loadings say how much of taste’s scale each other item moves per unit of Food (fresh 0.86, value 0.76), and Food’s variance, 0.83, is in taste units. The alternative is to fix the factor’s variance to 1 instead, which frees every loading. Neither choice changes the fit or the standardized results; the app’s Factor scaling option picks between them.

Standardized loading is a loading with both the factor and the item scaled to variance 1 – the correlation between item and factor – and the form to read and to report. The app prints it under Std. (completely); taste on Food is 0.77. Its square is the item’s R².

R² of an indicator is the share of an item’s variance the factor explains, the CFA name for a communality: 0.59 for taste, 0.33 for value, the lowest of the six. One minus it is the standardized residual variance.

Residual variance is the variance of an indicator that its factor does not explain – the CFA’s uniqueness – estimated as a parameter. taste’s is 0.58 in raw units, 0.41 standardized. A negative one is a Heywood case.

Factor covariance is the covariance between two latent variables, 0.30 for Food and Service in raw units, whose standardized form is the factor correlation of 0.45 (95% CI 0.28 to 0.62).

Free parameters are the numbers the model estimates from the data. The café CFA has 13: four loadings (the two markers are fixed), six residual variances, two factor variances and one covariance. The data supply 21 numbers – six variances and fifteen covariances – and the difference, 8, is the model’s degrees of freedom.

Model degrees of freedom are the number of observed variances and covariances minus the number of free parameters: the number of ways the model could have been wrong. With 8 df the café CFA makes 8 testable claims about the 21 numbers; a model with 0 df makes none and always fits, a saturated model. More df means a more falsifiable, more parsimonious model, and the parsimony-adjusted indices below reward it.

Identification is whether the data contain enough information to pin every free parameter to one value. A one-factor model needs three indicators; with two it has more parameters than data. A two-factor model is identified with two indicators each if the factors correlate. An unidentified model produces a warning, absurd standard errors or no solution at all, and the remedy is more indicators or fewer parameters, never a different estimator.

Convergence is the estimator reaching a solution – a set of parameter values at which the fit stops improving. A model that did not converge has no results worth reading; the app says so, and the usual causes are an unidentified model, a Heywood case or a bad starting point that a different marker variable fixes.

Estimator is the method that finds the parameter values – maximum likelihood by default, which assumes the indicators are roughly multivariate normal and is what this page’s CFA and SEM use; a robust variant that corrects the χ² and standard errors for non-normality; or a least-squares estimator for ordinal indicators. The café items are 1–5 answers treated as continuous here, which is common practice for five categories with a reasonable spread; the SEM manual’s CFA estimator section says when to switch.

Does the model fit? #

A CFA or SEM implies a correlation matrix – the one its loadings and paths would produce – and fit is the distance between that and the correlations observed. There is no single measure of the distance, so a model is judged on several indices that penalise different things, and the convention is to report the χ² test, CFI, TLI, RMSEA with its interval, and SRMR together.

Fit index is one such measure. Each has a rule of thumb attached, and the rules are guides, not laws: they were set by simulation on particular models, and a model can pass every one while misfitting in a way that matters, or fail RMSEA while being the best available description of the data. Read the indices together and beside the residual correlations.

Model fit indices is the results table that holds them, in the factor, CFA and SEM modules alike; the checkbox of that name prints it. For a regression model, whose fit is a different question, see model fit.

Chi-square test of model fit tests whether the model reproduces the data exactly – the null hypothesis is that it does. For the café’s two-factor CFA χ²(8) = 6.41, p = 0.60: the data do not contradict the model. For the one-factor model χ²(9) = 64.07, p < 0.001: they do. It is the one index with a p-value, and it is read the opposite way from most tests – a high p is good news – which the p-value page explains under which null. It also grows with the sample, so with a thousand cases almost every model is rejected, and the indices below exist to judge fit in degrees rather than as a verdict.

Saturated model is the model with as many parameters as data points – zero df – which reproduces the correlations exactly and fits perfectly by construction. It is the ceiling the χ² measures the fitted model against: the test statistic is the distance from the saturated model. On the café items three factors is that model: the CFI of 1.00 and SRMR of 0.002 it reads are arithmetic, not evidence.

Baseline model is the model at the other end: every indicator independent of every other, no factors at all. It is the floor the incremental indices measure from – the café’s baseline has χ² = 233.5 on 15 df – and CFI and TLI read the fitted model as a fraction of the way from that floor to the saturated ceiling.

CFI – the comparative fit index – is how far the model has come from the baseline towards perfect fit, from 0 to 1, with the χ² of each corrected for its df. The café’s two-factor model reads 1.00 (its χ² is below its df, so the correction puts it at the ceiling); the one-factor model reads 0.75. The usual thresholds are 0.95 for good fit and 0.90 for acceptable. It is the least sensitive of the indices to sample size, and the one most often used to compare nested models by its change – see ΔCFI.

TLI – the Tucker–Lewis index, also called the non-normed fit index – is CFI’s older sibling with a stronger penalty for free parameters, so it rewards a parsimonious model and can exceed 1 when the model over-fits its df; the café’s two-factor model reads 1.01, the one-factor 0.58. Same thresholds as CFI. The two usually agree; when TLI is well below CFI the model is buying its fit with parameters.

RMSEA – the root mean square error of approximation – is the misfit per degree of freedom, adjusted for sample size, from 0 upward: how badly the model fits the population, per claim it makes. Below 0.05 is close fit, up to 0.08 reasonable, above 0.10 poor. The café’s two-factor model reads 0.000 with a 90% interval of 0.000 to 0.071, the one-factor 0.175 (0.137 to 0.218) – the interval is the part to read, since a point estimate of 0 on 200 cases says less than an upper bound of 0.07. The app also gives p-close, the probability that the population RMSEA is below 0.05 (0.85 here). RMSEA is unreliable on models with very few df, where its interval is wide whatever the fit, and the app withholds it there.

SRMR – the standardized root mean square residual – is the average size of the residual correlations: how far, on average, the correlation the model implies is from the one observed. Below 0.08 is the usual threshold. The two-factor café model reads 0.034 – its residuals are all under 0.09, the largest 0.088 between value and slow – and the one-factor model 0.106, with residuals of 0.17 and 0.18 between items in the same trio that one factor cannot reach. It is the most direct of the indices, since it is the misfit itself and not a function of χ².

GFI – Jöreskog and Sörbom’s goodness-of-fit index – is the share of the observed variances and covariances that the model reproduces, from 0 to 1: an R² for the whole covariance matrix, where 1 means the matrix the model implies is the one observed. The café’s two-factor model reads 0.989, the one-factor model 0.885. It makes no charge for the parameters spent reaching that fit – AGFI and PGFI are the two ways of charging for them – and it rises with the sample size whatever the model, which is why it has fallen out of use as a verdict beside CFI and RMSEA; 0.90 was the traditional bar.

AGFI – the adjusted goodness-of-fit index – is GFI with a charge for the parameters spent reproducing the data: the share GFI leaves unreproduced is scaled up by the ratio of the data’s variances and covariances to the model’s degrees of freedom, 21 to 8 for the café’s two-factor model. That model reads 0.97, the one-factor model 0.73. It inherits GFI’s rise with the sample size, and so its fall from use as a verdict; 0.90 was the traditional bar.

NFI – Bentler and Bonett’s normed fit index – is the share of the baseline model’s χ² that the fitted model removes, $({\chi }_0^2-{\chi }_m^2)/{\chi }_0^2$, from 0 to 1. The café’s two-factor model removes 97% of the baseline’s 233.5 (NFI 0.97), the one-factor model 73%. It makes no correction for degrees of freedom, so adding a parameter can only raise it, and it runs low in small samples even for a correct model – the two faults that the indices below, and CFI, were built to fix. The usual bar is 0.95.

RFI – Bollen’s relative fit index – is NFI computed on χ² per degree of freedom: how far the model’s χ²/df has come down from the baseline’s, 15.6 on the café’s baseline to 0.80 for the two-factor model, an RFI of 0.95; the one-factor model reads 0.54. Dividing by df charges the model for its parameters, so RFI sits below NFI and can fall when a parameter is added. TLI is the same ratio with the χ²/df a correct model would show, 1, subtracted from the baseline’s.

IFI – Bollen’s incremental fit index – is NFI with the fitted model’s df subtracted in the denominator, $({\chi }_0^2-{\chi }_m^2)/({\chi }_0^2-d{f}_m)$: the improvement over the baseline measured against the improvement a correct model would be expected to make. That removes most of NFI’s small-sample dip and, as with TLI, lets it pass 1 when the model’s χ² falls below its df: the café’s two-factor model reads 1.01, the one-factor model 0.75.

PNFI – the parsimony normed fit index of James, Mulaik and Brett – is NFI multiplied by the share of the baseline’s degrees of freedom the model keeps, $d{f}_m/d{f}_0$, so a model that spends every df on parameters scores 0 however well it fits. The café’s two-factor model keeps 8 of 15 and reads 0.52; the one-factor model keeps 9 and reads 0.44. It has no accepted threshold: it compares models fitted to the same data, higher meaning more fit per parameter spent – and its parsimony weight can outrun the fit, as PGFI shows.

PGFI – the parsimony goodness-of-fit index of Mulaik and colleagues – is GFI multiplied by the model’s degrees of freedom as a share of the data’s variances and covariances, 8 of 21 for the café’s two-factor model. That model reads 0.377 and the one-factor model 0.379: the model every other index rejects scores a hair higher, because the one df it saves outweighs the fit it loses. That is the parsimony indices’ limit – they trade fit against simplicity at a fixed rate nobody calibrated – and why they are read beside the other indices, never as a verdict.

Sample-adjusted BIC is the BIC with a lighter penalty per parameter, for comparing models on the same data as the app’s SEM tables do: lower is better, and a difference of 10 is decisive. Alongside it the tables carry the plain AIC and BIC; the café’s two-factor CFA beats the one-factor by 56 on AIC and 52 on BIC.

Model comparison across factor counts is the EFA reading of the fit indices: the same χ², RMSEA, CFI, TLI, SRMR and information criteria computed for every candidate count in turn, one row per count, and read for the row where they settle rather than for any one row’s verdict. The café’s rows run: one factor χ²(9) = 64.7, p < .001, RMSEA 0.18, CFI 0.74, SRMR 0.10 – poor on every index; two factors χ²(4) = 1.05, p = .90, RMSEA 0 with an interval of 0 to 0.045, CFI 1.00, TLI 1.05, SRMR 0.01 – as good as an index gets; three factors, the saturated count at 0 df, reproduces the correlations by construction and has no fit to grade. Two is the row, and the shape is the usual one – a leap in fit at the true count and nothing to gain past it. The χ² and the indices built on it come from the maximum likelihood discrepancy at whatever solution the extraction reached, so under any other method they describe the fit rather than test it and read a little worse than an ML fit would, which the app says beneath the table. Fit is one voice beside the retention rules: a count that fits and cannot be read is not a finding, and the factor analysis manual reads the table row by row.

EBIC – the empirical BIC – is the BIC built from a different χ²: not the likelihood’s, which belongs to maximum likelihood and is only borrowed under any other extraction, but the one implied by the residual correlations the solution actually leaves behind, which every extraction can be judged on. The two count misfit differently – the likelihood’s χ² counts a residual for more the better the items involved are measured, the residual χ² counts every residual alike – so their minima can land on different rows, and when they do EBIC tends to favour the larger count. On the café items they agree: BIC reads 17.0 for one factor and −20.1 for two, EBIC −4.0 and −20.8, both at their minimum on the two-factor row. ESABIC is EBIC with the sample-size adjustment, 24.5 and −8.1 on the same two rows. A disagreement between the two minima is a genuine ambiguity in the data, settled by parallel analysis and by which solution can be read, not by picking the column one prefers.

Chi-square difference test compares two nested models – one is the other with some parameters fixed – by the difference in their χ² on the difference in their df: a likelihood-ratio test for structural models. The one-factor café model is the two-factor model with the factor correlation fixed at 1, so Δχ² = 64.07 − 6.41 = 57.65 on 1 df, p < 0.001: freeing the correlation improves the fit far beyond chance, and two factors it is. It is the test behind every step of the invariance cascade. Under a robust estimator the two χ² are scaled and the difference is not a plain subtraction; the app does the right arithmetic.

Modification indices say how much χ² would fall if one fixed parameter were freed – a loading the model sets to zero, a covariance between two residuals – one index per candidate. The café CFA’s largest are 4.15 for letting fresh load on Service and 4.15 for correlating the residuals of taste and value, each worth about one df of χ² and neither worth acting on: an index below about 10 is noise, and a change made because the index says so is a change made to fit this sample, which the next sample will not repeat. Use them to find one clear misspecification with a substantive reason – two items with the same wording, a known secondary loading – and report the change as post hoc.

EPC is the expected parameter change beside the MI – the value the freed parameter would take (0.32 for fresh on Service, 0.20 standardized) – which says whether the improvement in χ² would come from a parameter of any size. A large MI with a tiny EPC is a sample-size artefact.

Residual correlations are the observed correlations minus the ones the model implies, item pair by item pair: the model’s misfit, located. taste and fresh correlate 0.511 in the data and 0.523 under the two-factor model, a residual of −0.012, and the largest of its fifteen is 0.088, between value and slow; under the one-factor model the residuals inside each trio are 0.09 to 0.18 and across the trios −0.06 to −0.14 – slow’s own with the opposite sign, since it runs the other way – which is the signature of a missing factor – pairs the model under-predicts, pairs it over-predicts, in blocks. A residual above 0.10 in absolute value is worth a look; the app counts them and prints the matrix when Residual correlations is checked. The two-factor EFA of the café items leaves every residual within 0.03 and none above 0.05; the two-component PCA of the same items leaves 8 of the 15 above it, since a component carries each item’s own variance rather than reproducing the correlations. psych’s two fit shares summarise the same gap: the share of the correlations’ sum of squares the solution reproduces reads 0.999 for the two factors and 0.85 for the two components, while the whole-matrix share, diagonal included, reads 0.78 for almost any solution and is not variance explained.

Is the scale sound? #

A model that fits says the items hang together the way the model claims. Whether they hang together enough to be one scale, and whether the two scales are really two, are the questions of convergent and discriminant validity, and both are answered from the CFA’s standardized loadings.

Convergent validity is the extent to which a factor’s indicators agree – that they measure one thing and measure it well. Its measures are the size of the standardized loadings (all above 0.5 in size is the usual ask; the café’s run from 0.57 to 0.77, slow’s negative), the AVE and the composite reliability.

Average variance extracted is the mean of the squared standardized loadings of a factor’s indicators: the share of the items’ variance the factor explains, on average. Food’s is 0.46 and Service’s 0.41 – below the conventional 0.50, which says that more than half of each item’s variance is noise, as it commonly is for three short questionnaire items. The rule is a guide; a three-item scale with loadings around 0.65 and an AVE of 0.45 is usable, and the number to weigh it against is the composite reliability.

Composite reliability (CR) is the reliability of the sum of a factor’s indicators, computed from their loadings rather than from their correlations as Cronbach’s α is: the share of the sum’s variance that is the factor. Food’s is 0.71 and Service’s 0.67 – the same numbers the app prints as McDonald’s ω, which is the same coefficient under another name. Service’s is the sum with slow reverse-scored: an item loading against the rest of its factor would cancel the others in the sum, so the app reverses it for the reliability coefficients alone and names it with its loading in a note under the table. Above 0.7 is the usual ask, and a CR above 0.6 with an AVE below 0.5 is read as adequate convergent validity. The reliability analysis manual covers the family of coefficients.

Discriminant validity is the extent to which two factors are distinct – that Food and Service are two things and not one thing measured twice. A factor correlation near 1 fails it outright; the two criteria below are the graded tests, and the app’s Discriminant validity option prints both.

Fornell–Larcker criterion asks that each factor share more variance with its own indicators than with any other factor: the square root of its AVE must exceed its correlations with the other factors. Food’s √AVE is 0.68 and Service’s 0.64, both above their correlation of 0.45, so the criterion is met.

HTMT – the heterotrait–monotrait ratio – is the average correlation between items of different factors divided by the average correlation between items of the same factor, each correlation taken in size so that a reverse-keyed item counts by its strength. The app averages them as geometric means – the HTMT2 form, which stays consistent when the items load unequally – so for Food and Service the across-trio correlations average 0.17 and the within-trio ones 0.45 and 0.40, giving 0.41; below 0.85 is distinct, and above 0.90 the two factors are one. It is the more sensitive of the two criteria and the one to report when they disagree.

The same scale in two groups #

Comparing Riverside’s satisfaction with Station’s assumes the six items measure satisfaction the same way at both branches. That is an assumption with a test, and the test is a sequence of CFA models fitted to the two groups at once with more and more of their parameters forced equal.

Measurement invariance is the property that a scale measures the same construct on the same metric across groups, occasions or languages, so that a difference in scores is a difference in the construct and not in how the items behave. It is tested in levels, each nested in the last, and a group comparison of means is licensed at the scalar level. The café’s cascade by branch runs: configural χ²(16) = 23.55, p = 0.10; metric Δχ²(4) = 2.22, p = 0.70; scalar Δχ²(4) = 1.39, p = 0.85; strict Δχ²(6) = 25.04, p < 0.001. The scale is invariant through the scalar level and not beyond, which is enough to compare the branches’ latent means. The SEM manual’s invariance section runs the cascade and reads its table.

Configural invariance is the first level: the same model – the same items on the same factors – fitted in each group with every parameter free. It asks only whether the structure is the same shape, and it is the baseline the later levels are compared with. The café’s configural model fits acceptably, CFI 0.97, RMSEA 0.069.

Metric invariance – also called weak invariance – adds equal loadings across groups: each item relates to its factor equally strongly at both branches, so a unit of Food means the same thing in each. It licenses comparing the factors’ correlations and regression paths between groups. The café passes: Δχ²(4) = 2.22, p = 0.70, ΔCFI +0.008.

Scalar invariance – strong invariance – adds equal intercepts: a member with the same food satisfaction gives the same expected answer to taste at either branch. It licenses comparing the latent means, since without it a mean difference could be an item behaving differently rather than the construct differing. The café passes: Δχ²(4) = 1.39, p = 0.85, ΔCFI +0.012.

Strict invariance adds equal residual variances: the items are equally noisy in both groups. It is rarely required and rarely met; the café fails it, Δχ²(6) = 25.04, p < 0.001, ΔCFI −0.085, because Station’s answers are more spread than Riverside’s on every item, significantly so on five – the branch was built more heterogeneous. Failing strict invariance does not undo the levels below it.

Structural invariance goes past measurement to the latent variables themselves – equal factor variances and covariances across groups – and is a hypothesis about the constructs, not about the scale; it is judged on its own terms rather than as a licence. The café fails it too, Δχ²(3) = 11.68, p = 0.009.

ΔCFI is the change in CFI from one invariance level to the next, the practical criterion used beside the Δχ² because the χ² test rejects trivial differences in large samples: a drop of more than 0.01 is the usual sign that a level fails. The café’s drops are +0.008, +0.012 and −0.085 – the last is the strict level failing by both criteria. The app also reports ΔRMSEA (a rise above 0.015 fails) and ΔSRMR.

Partial invariance is what is claimed when a level fails because of one or two parameters: those are freed, the rest stay equal, and the comparison proceeds with the caveat. The app’s score tests name the parameters that would do the most to restore fit – at the strict level the residual variance of value, χ²(1) = 8.18, p = 0.004, is the largest offender. Partial invariance is exploratory and should be reported as such.

Latent mean is the mean of a latent variable in a group, on the scale the marker variable set, with one group fixed at 0 as the reference. It is estimable once scalar invariance holds. With Riverside at 0, Station’s Food mean is +0.11 (SE 0.15, p = 0.45) and its Service mean −0.34 (SE 0.13, p = 0.008): Station’s members are as satisfied with the food and less satisfied with the service, which is the pattern the data were built with. Divided by a latent standard deviation the difference becomes a Cohen’s d on the latent scale, readable across scales – Service’s is −0.40.

Item response theory #

Factor analysis treats an item’s answers as a continuous variable; item response theory treats them as what they are – categories – and models the probability of each answer as a function of the person’s position on the trait. The reward is a scale on which items and people sit together, an estimate of how precisely each person is measured, and a way to ask whether an item works the same for everyone. This page fits the six café items as one trait, “satisfaction”, to show what the numbers mean, knowing from the unidimensionality check that the scale is two-dimensional, and shows below how that shows up in the diagnostics; the honest analysis fits Food and Service separately, or as a two-dimensional model – multidimensional IRT, below, which the IRT manual runs.

Item response theory is the family of models in which the probability of a response to an item depends on a person parameter, the trait, and one or more item parameters – how hard the item is, how sharply it separates people. Where classical test theory characterises a test as a whole, IRT characterises each item, and its person estimates do not depend on which items were answered, so that two forms of a test can be put on one scale.

Classical test theory is the older framework in which a person’s score is the sum of the item scores, the test’s reliability is a single number, and an item’s difficulty is the share who answered it correctly. It is simple and adequate for many purposes, and it is what reliability analysis does; its limitation is that every one of its numbers depends on the sample it was computed in – an item is “easy” only for this group – which IRT removes.

θ – theta – is the person’s position on the trait, the latent variable as IRT names it, on a scale with mean 0 and standard deviation 1 in the sample: a member at θ = 1 is one SD more satisfied than average. The café members run from −2.53 to 1.99, and each estimate carries its own standard error, 0.44 to 0.63 – a person’s θ is measured, not counted. The app summarises the estimates under Person ability estimates, labels the θ axis of its plots Ability (θ) whether the instrument is a test or a questionnaire, and writes each person’s θ to the dataset on request.

Item characteristic curve is the plot of the probability of a response against θ for one item: an S-shaped curve for a right-or-wrong item, and for a 1–5 item a set of curves, one per answer category, each peaking where that answer is the most likely. Its position along θ is the item’s difficulty, its steepness the discrimination. Two items with the same curve are interchangeable; the plot is the fastest way to see how a set of items covers the trait.

Difficulty is the point on θ where an item’s response turns over – for a right-or-wrong item, the θ at which the chance of a correct answer is 50%; for a 1–5 item, the θ at which each successive answer becomes more likely than not. It is on the same scale as θ, so it says who the item is for: taste’s four thresholds are −2.06, −0.95, −0.09 and 0.89, so a member of average satisfaction has a 55% chance of agreeing (4 or 5) with “the food tasted good” and a 13% chance of strongly agreeing; friendly’s thresholds are −3.51, −2.43, −1.00 and 1.01, so the same average member agrees with “the staff were friendly” 69% of the time – it is an easier item to agree with, which is what a lower difficulty means.

Threshold is a difficulty for one step of a polytomous item – the θ at which answering k or above becomes more likely than answering below it. A 1–5 item has four, in increasing order; the app heads them Threshold 1 to Threshold 4, or b1 to b4, each with its standard error. Thresholds far below the sample – friendly’s −3.51 – mark an answer category almost nobody used.

Discrimination is how sharply an item separates people on either side of its difficulty: the slope of the characteristic curve at its steepest. A high value means a small difference in θ makes a large difference in the expected answer, so the item is informative; a low one means the answer is nearly the same for everyone. On the six café items taste discriminates 2.12 (SE 0.42), fresh 1.71, value 1.20 and the three service items 0.79 to 0.83 – the food items dominate the single trait, which is the first sign that the scale is two-dimensional rather than one. Values above 1.35 are high, 0.65 to 1.34 moderate, below 0.65 low, and a negative one is a reverse-keyed item left out of Negatively keyed items – slow, picked there, is fitted reversed and discriminates positively.

Intercept (d) is the parameterisation the estimator works in – the item’s easiness at θ = 0, one per threshold – from which the difficulties ($b=-d/a$) are derived and printed alongside.

Guessing is the lower asymptote of a characteristic curve: the chance of a correct answer from a person with no ability at all, as on a multiple-choice item with four options, where it is about 0.25. It is the third parameter of the 3PL model and has no meaning on a rating item.

Rasch model is the simplest IRT model: every item has its own difficulty and all share one discrimination, so the raw sum score carries all the information about θ and items and people sit on one scale with equal spacing. It is the model of choice when the items were written to be equally good indicators and the sum score is what will be reported; its price is that it cannot describe an item that discriminates much better than the others, and its fit statistics (infit and outfit) say when that happens. The one-parameter logistic model, 1PL, is the same model; for ordered categories it becomes the partial credit model.

2PL is the two-parameter logistic model: each item has its own difficulty and its own discrimination. It is what the café items were fitted with (its polytomous form, the graded response model), and the spread of discriminations from 0.79 to 2.12 is what the Rasch model would have had to ignore.

3PL adds a guessing parameter to the 2PL, for multiple-choice tests. It needs large samples – the guessing parameter is hard to estimate – and is never right for a rating scale.

Graded response model is the 2PL for ordered categories: one discrimination per item and one threshold per step between adjacent answers, modelling the probability of answering at or above each category. It is the model for Likert items and the one this page’s numbers come from; the models after it are its alternatives for ordered answers.

Partial credit model is the Rasch model for ordered categories, built from the steps between adjacent answers: each step asks only whether a person who gave one of two neighbouring answers gave the higher one, and its difficulty is the θ at which the two are equally likely. Where the graded response model cuts one underlying scale into categories, the partial credit model treats each step as a small right-or-wrong item of its own, all steps of all items sharing one discrimination, so the sum score still carries all the information about θ. The app fits it when Rasch (1PL) is chosen for items with more than two categories. On the six café items it spends 25 parameters where the graded model spends 30; it fits a little worse by AIC (3500.8 against 3495.6) and better by BIC (3583.2 against 3594.6), which charges more for each parameter.

Generalized partial credit model gives each item of the partial credit model its own discrimination, as the 2PL does the Rasch model’s: the same adjacent-category steps, each item separating people as sharply as its data say. It is the graded model’s usual alternative for rating items, and the two describe the same data in different terms and rarely disagree in substance: on the café items it fits almost as well (AIC 3497.2 against 3495.6, with the same 30 parameters). Its discriminations are not comparable with the graded model’s, since each is the slope of a different curve – taste’s is 1.16 here and 2.12 there.

Disordered steps are step difficulties out of order – a step lower than the one before it, which the adjacent-category models allow and the graded model cannot produce. They mean that the category between the two steps is never the single most likely answer at any θ: whoever might give it is always likelier to give a neighbour. On the café’s friendly the step from 1 to 2 sits at −0.96 and the step from 2 to 3 at −2.45, so “disagree” is never the most likely answer – 15 members gave it, about as many as gave 1 and fewer than half as many as gave 3 – and slow’s last two steps are reversed the same way. The café was generated from cut points in order, so a reversal marks a narrow category, not a broken item; Rasch practice reads one as a category respondents do not use as intended and considers merging it with a neighbour, which is a decision about the response format rather than about the item.

Rating scale model – Andrich’s – is the partial credit model with one set of step difficulties shared by every item, each item shifted along θ by a location of its own: the items differ in how easy they are to agree with, not in how their answer categories are spaced. It is the model for a questionnaire whose items share one response format and are meant as interchangeable indicators, and its price is the Rasch model’s – one discrimination for all. On the café items it needs 10 parameters to the graded model’s 30 and has the lowest BIC of the ordered-category models on this page, 3525.8; its shared steps are −1.21, −0.77, −0.19 and 0.41, and its item shifts order the items exactly as their means do, from friendly, the easiest to agree with (mean 3.79), to value, the hardest (3.34).

Generalized rating scale model gives each item of the rating scale model its own discrimination back: one shared pattern of thresholds, shifted item by item, on the cumulative categories of the graded response model. On the café items it has the lowest AIC of the models on this page, 3488.4 against the graded model’s 3495.6, with 15 parameters to its 30. That is the café’s generating model showing through: every item’s cut points were set almost the same distance apart (0.7 to 1.0) and shifted item by item, with loadings that differ – the structure this model assumes – so the graded model’s freedom to space each item’s thresholds on its own buys less likelihood than its fifteen extra parameters cost.

Nominal response model drops the order: every answer category gets its own slope on θ, and the data say which categories go with a higher θ and in what order. It is for answers with no known order – the options of a multiple-choice question, whose wrong answers may be wrong to different degrees – and a check on a rating scale’s order: slopes that come out in category order confirm it. On the café items taste’s slopes run −2.34, −0.87, −0.13, 0.93 and 2.41 from “strongly disagree” to “strongly agree”, in order, while quick’s first two are −0.71 and −0.76, so answering 1 or 2 to “my order came quickly” says the same about a member’s satisfaction. The flexibility costs 48 parameters on six items, and the café’s AIC, 3517.3, is higher than that of any model that assumes the order – the order was there to be used.

Sequential response model treats an answer as the last of a series of steps taken in order: to reach category 3 a person passes step 1, then step 2, and fails step 3, each step a right-or-wrong item attempted only once the one before it is passed. It fits responses produced that way – the level reached in a staged exam, the stage a course was completed to – rather than a degree of agreement chosen in one go, which the graded model describes. On the café items it fits worse than the graded model (AIC 3503.2 against 3495.6, with the same 30 parameters) while giving much the same discriminations (taste 2.05 against 2.12): agreement ratings carry no sign of a staged process, as they should not.

Unfolding is the other way an answer can follow the trait: agreement peaks where the person’s own position meets the statement’s and falls away on both sides, so the item’s characteristic curve is single-peaked rather than rising. Every model above assumes the opposite, a dominance item – the higher θ, the more the person agrees, as with “the food tasted good” – which suits a question about how much of something a person has: satisfaction, ability, anxiety. Unfolding suits a statement that stakes out a position between two poles, which a person endorses only when it matches their own: an attitude, a preference, a political stance. The statement’s position on the θ scale is its ideal point, and the app fits such items with the Ideal point model for right-or-wrong answers and GGUM, the generalized graded unfolding model, for rating ones. The café’s coffee survey (cafe-coffee.csv, generated from GGUM, its spec beside it) asked 200 members how far they agree, 1 to 4, with six statements running from “I like milky drinks where the coffee is barely there” to “I want my coffee as strong and black as it comes”, so θ is how strong a member likes their coffee. GGUM places the six statements at −1.54, −0.87, −0.20, 0.29, 0.91 and 1.55, where they were planted at −1.5, −0.9, −0.3, 0.3, 0.9 and 1.5. filter – “a regular filter coffee suits me” – is agreed with (3 or 4) by 91% of the members whose taste sits at its ideal point, by about 64% of those 0.8 to either side, and by 14% of those who like their coffee milky (θ = −2) and 6% of those who like it black (θ = 2). The graded response model cannot draw that curve and makes the best of a rising one: it gives milky and latte negative discriminations, −3.11 and −1.27, as if they were reverse-keyed, espresso and black positive ones, 1.72 and 1.89, and filter −0.18 (SE 0.15) – a good statement that it reads as measuring nothing, because members on both sides of it disagree alike. GGUM fits far better despite its six extra parameters (AIC 2936.7 against 3048.4, BIC 3035.7 against 3127.5). A statement at the edge of the sample looks like a dominance item – black’s curve is still rising at θ = 2, where 80% agree – so the evidence for unfolding is in the middle statements, and a scale whose items all sit near its two ends cannot tell the two models apart.

Item information is how much an item contributes to the precision of θ at each point of the scale: a function of θ that peaks near the item’s difficulty and is taller the higher its discrimination. A test made of items with all their information at θ = 0 measures average people precisely and extreme ones badly.

Test information is the sum of the item information functions – the precision of the whole test along θ. The six café items give most of their information between θ = −2 and θ = 1, peaking at 3.27 near θ = −0.65, and fall to 1.55 by θ = 2: the survey measures the mildly dissatisfied and the average well and the delighted poorly, because nearly every threshold sits below 1. The standard error of θ is its reciprocal square root.

SE(θ) is the standard error of a person’s θ: 1 divided by the square root of the test information at their θ. Where the café survey’s information is 3.27 the SE is 0.55, and at θ = 2 it is 0.80 – the same person measured to about ±1.1 or ±1.6 at 95% confidence. Unlike a classical reliability, it is a precision per person, and it is what makes IRT the framework for adaptive testing.

Marginal reliability is the classical-style summary of an IRT test’s precision: one number for the whole sample, the share of θ’s variance that is not measurement error, averaged over the sample’s distribution. The six café items give 0.75; a reliability analysis’ α or ω on the same items is the CTT counterpart and usually close.

Empirical reliability is the same summary computed from the estimated θs and their standard errors directly – 0.75 here, the same to two decimals. The two diverge when the sample’s θ distribution is far from the one assumed.

Person separation is the ratio of the true spread of θ in the sample to its measurement error – 1.71 here – and the number of strata it implies, 2.6, is how many distinct levels of the trait the test can tell apart: about three, from dissatisfied through average to satisfied. Two strata or fewer is a test that sorts people into high and low and no finer.

Test targeting is how well the items’ difficulties are placed for the sample’s θs: the difference between the mean item location and the mean person location, in θ units. The café items sit 0.83 below the members on average – the survey is “easy”, in that agreement is the common answer – so the precise part of the scale is below the middle of the sample, as the test information curve shows.

Wright map puts the people and the item thresholds on one θ axis, side by side: a histogram of θs on one side, each threshold as a mark on the other. Where the thresholds bunch, the test measures finely; where the people bunch and no threshold sits, they cannot be told apart. On the café’s map the thresholds crowd below −1 and thin out above 1, and the members centre on 0.

Item fit asks whether an item’s answers follow its characteristic curve – whether people who should agree, given their θ, do. Its statistics are a χ² comparing observed and expected responses in slices of θ, printed as S-χ² with its df and p (none of the café items rejects; taste’s is the closest at χ²(27) = 38.6, p = 0.07), and the infit and outfit mean squares that Rasch tradition reads.

S-χ² is the item-fit χ² – a significant value marks an item whose answers do not follow the model’s curve, and with many items the p-values are adjusted. Read it as item fit.

Infit is a mean-square fit statistic weighted towards the people the item is targeted at. It has an expected value of 1: above about 1.5 the item is noisier than the model expects (an item that measures something else, or badly worded); below about 0.5 it is more predictable than the model expects. The café’s taste reads 0.67, the lowest of the six – an item that is too good for a single-trait model, because it dominates the trait, and the second sign of the two dimensions. The app prints it as Infit MNSQ, with a standardized form, Infit ZSTD, that reads as a z.

Outfit is the unweighted mean square, read on the same scale as infit and more sensitive to unexpected answers from people far from the item’s difficulty – a strongly satisfied member who disagreed that the food tasted good. taste reads 0.63. Printed as Outfit MNSQ and Outfit ZSTD.

Person fit asks the same question of each person: whether their pattern of answers is plausible given their θ. A member who strongly agrees the food was fresh and strongly disagrees it tasted good has a pattern the model did not expect, and the statistic – a Zh below −2 – flags them. Three café members are flagged where 4.6 would be by chance, so the sample is clean; a flagged person on a test is a candidate for guessing, cheating or inattention, on a survey for careless responding.

Local independence is the assumption that once θ is accounted for, the items are unrelated: what two items share is the trait and nothing else. It is the IRT form of unidimensionality, and it fails when two items share wording, content or a second dimension.

Local dependence is its violation, found by correlating the residuals of each pair of items after the model – Yen’s Q3. A Q3 above 0.2 in absolute value is the usual flag. On the six café items as one trait, friendly–quick reads 0.28, quick–slow 0.28 and friendly–slow 0.21 – the service items are related beyond what one satisfaction explains – and taste–quick reads −0.33, the food and service items related less: the signature of the second dimension, seen from inside the model.

Yen’s Q3 is the local-dependence statistic under its usual name – read it as local dependence; in a descriptive table Q3 is the third quartile instead.

Multidimensional IRT – MIRT – places each person on several traits at once, each item depending on one or more of them, as in a factor analysis, but with the answers modelled as categories. It comes in factor analysis’ two forms: exploratory, where every item may load on every dimension and a rotation makes the solution readable, and confirmatory, where each item is assigned to its dimensions in advance. On the café items a confirmatory two-dimensional graded model – Food on taste, fresh and value, Service on friendly, quick and slow – fits far better than the single trait this page uses (AIC 3446.6 against 3495.6, BIC 3548.8 against 3594.6). A multidimensional fit reads the items as answered, whatever the keying picker holds, so slow loads negatively: the standardized loadings are 0.86, 0.68 and 0.61 on Food and 0.65, 0.71 and −0.64 on Service, where the café was generated with 0.75, 0.70 and 0.60 and 0.75, 0.65 and 0.60, and the two traits correlate 0.46 against a planted 0.5. An exploratory fit of two dimensions with an oblimin rotation finds the same split unaided – each item loading 0.59 to 0.84 in size on its own dimension and no more than 0.15 on the other – at a higher BIC (3564.9), having spent four more parameters to find what the confirmatory model assumed.

Compensatory model is a multidimensional model in which the traits add up: the response depends on a weighted sum of the person’s positions, so a high value on one trait makes up for a low one on another. The standard MIRT models, the café’s two-dimensional one included, are compensatory, which suits an item a person can answer well by more than one route.

Partially compensatory model multiplies instead: the chance of a correct answer is the product of one success probability per trait, so every trait the item needs has to be there – a mathematics word problem needs both the reading and the arithmetic, and no amount of one makes up for the lack of the other. It is defined for right-or-wrong items, and it needs the traits each item draws on declared in advance, in a confirmatory model.

Differential item functioning is an item behaving differently for two groups of people who have the same θ: a question that is harder for one branch’s members than the other’s at equal satisfaction. It is a property of the item, not of the groups – a group that is genuinely more satisfied shows a difference in θ, called impact, and not DIF – and it is the IRT form of a measurement invariance failure, tested item by item. Fitted to the three Food items by branch, fresh and value flag at p = 0.035 each and taste does not (p = 0.08); but the app’s effect sizes are 0.06 and −0.11 on the expected-score scale, negligible on a 1–5 item, and a DIF with a negligible size is reported and not acted on. On the six-item single-trait model five items flag, because the branches differ on Service and the model has no Service to put the difference in – a wrong model manufactures DIF.

Uniform DIF is a difference in difficulty only: the item is harder for one group at every θ, which is what the Food items’ flags above are.

Non-uniform DIF is a difference in discrimination as well, so the item favours one group at low θ and the other at high θ; the app tests it separately, and none of the Food items shows it (p = 0.30 to 1.00).

Anchor items are the items assumed free of DIF that put the two groups on one θ scale, so that the others can be tested against them. The app picks them by testing every item against all the others and keeping the clean ones, or takes a designated set; a scale with no clean anchor cannot be tested for DIF at all.

Expected score is the answer the model predicts for a person at a given θ – the sum over categories of each answer times its probability – and the expected total score is its sum over items: the curve that maps θ back onto the raw-score scale of the survey. A member at θ = 0 is expected to score about 21 of 30 on the six items, slow counted reversed.

Raw score is the plain sum of a person’s item answers, the negatively keyed ones reversed – 6 to 30 on the café survey. Under the Rasch model it carries all the information about θ; under a 2PL it does not, since two people with the same raw score may have answered differently discriminating items. The app’s score-conversion table maps each raw score to a θ and its standard error – 6 maps to −2.86 – so a report can be written in either currency.

EAP – the expected a posteriori estimate – is the app’s default θ score: the mean of the person’s likelihood combined with the population distribution, which pulls extreme estimates a little towards the centre and gives every response pattern a finite score, including all-1s and all-5s, where maximum likelihood would give ±∞. Its standard error is smaller than ML’s for the same reason. The alternatives – MAP, MLE and WLE – follow; the scoring method setting picks among them.

Maximum a posteriori (MAP) is the θ score at the peak of the person’s posterior – the likelihood of their answers combined with the population distribution – where EAP takes its mean. Like EAP it is finite for every pattern and pulled towards the centre, and the two are usually close: the café members’ MAP scores run from −2.50 to 1.90 where their EAP scores run from −2.53 to 1.99, and the two sets spread with an SD of 0.86 and 0.87.

MLE – the maximum likelihood estimate – is the θ at which the person’s answers are most probable, from the answers alone: no population distribution, so no pull towards the centre and nothing assumed about the sample. Its cost is at the edges. It is biased outwards – a high scorer is estimated higher still – its standard error is larger, and for an extreme response pattern it does not exist, since the likelihood keeps rising as θ goes to ±∞. The café members’ MLE scores spread with an SD of 1.19 and reach down to −3.87, with a standard error of 0.95 there, where EAP stops at −2.53.

WLE – Warm’s weighted likelihood estimate – is the MLE with a correction that removes its outward bias: it maximises the likelihood weighted by the square root of the test’s information. It keeps what MLE has – no population distribution, no shrinkage – and gains a finite score for the extreme patterns. Its scores spread almost as widely as MLE’s, an SD of 1.16 on the café items, and it is the usual choice in Rasch practice and wherever a score should not depend on the population it was estimated in.

Shrinkage, of a θ score, is the pull of a Bayesian estimate – EAP or MAP – towards the centre of the population distribution, strongest for the people the test measures worst. It makes each score a better guess at that person’s θ on average, and it makes the scores less spread than the trait: the café’s EAP scores have an SD of 0.87 on a trait whose SD is 1, which is the square root of the marginal reliability, 0.75 – shrinkage takes out the share of the trait’s variance the six items cannot measure. A spread or a group difference computed from shrunk scores is understated, so an extreme group looks less extreme; an MLE or WLE score does not shrink and carries its error as extra spread instead (SD 1.19 and 1.16 on the café).

Extreme response pattern is a set of answers all in the lowest or all in the highest category – every item answered 1, or every item 5. It says only that the person is beyond the items’ reach, not how far: MLE has no finite estimate, WLE gives a large one with a large standard error, and EAP and MAP give a finite one that the population distribution holds in. Two café members gave the most satisfied answer to all six items – 5 to the five positively worded ones and 1 to slow: EAP puts them at 1.99, MAP at 1.90, WLE at 3.63 with a standard error of 1.69 – which says how little six items can tell about someone at the top of the scale – and MLE leaves them unscored. The more such patterns a sample holds, the more the choice of score matters, and the more the test needs items at that end of the scale.

Nonparametric IRT: Mokken scales #

The models above assume a shape – the logistic curve – for how the chance of each answer rises with θ, and estimate its parameters. Mokken scale analysis assumes only that the curve rises, and asks whether the items behave that way: whether their answers can be added into a score that orders people on the trait. It fits no model and needs no large sample, which is why the IRT manual runs it before a parametric fit is committed to. The numbers below are the six café items as one scale, with slow picked under Negatively keyed items so that the diagnostics read it reversed, as the fits above do.

Nonparametric IRT is item response theory without a formula for the item characteristic curve: the curve is only required to have a property – never to fall, or never to cross another item’s – and the data are checked for it. What it gives up is a θ on an interval scale and item parameters to go with it; what it keeps is the ordering of people by their sum score, which is what most questionnaires are used for. A set of items that fails a nonparametric check fails the parametric models built on the same assumption, so the check is the cheaper first look.

Guttman error is a pair of answers that contradicts the items’ order: agreeing with the harder item of a pair while disagreeing with the easier one. In a perfect scale – a Guttman scale – nobody makes one: everyone who agrees that the food is good value also agrees that it tasted good, since value is the harder of the two to agree with. Real items make them, and counting them against the number two unrelated items would produce is what Loevinger’s H does.

Loevinger’s H – the scalability coefficient – is one minus the ratio of the Guttman errors the data make to the number expected if the items were unrelated: 1 is a perfect scale, 0 items that share nothing, and a negative value an item that runs against the others, usually a reverse-keyed item left unreversed. It comes in three forms. H_ij is one item pair’s – taste and fresh scale together at 0.53, value and slow barely at 0.07; H_i is one item’s against all the others; and H is the whole set’s. The whole set is read in Sijtsma and Molenaar’s bands – under 0.3 not a scale, 0.3 to 0.4 weak, 0.4 to 0.5 moderate, 0.5 and above strong – and an item belongs in a scale at an H_i of 0.3 or more. The six café items give H = 0.299 (SE 0.034), with item values from 0.24 for value to 0.35 for fresh: as one scale they fall just short, value (0.24) and slow (0.27) the two items below the bar – the two dimensions again, seen as a food item and a service item that scale poorly with the other trio.

Mokken scale is a set of items whose every pair has a positive H_ij and whose every item has an H_i at or above a lowerbound c, conventionally 0.3: a set whose sum score orders people on one trait under the monotone homogeneity model. The six café items as one set are not one, by the numbers above; Food’s three and Service’s three each are, as automated item selection finds.

Monotone homogeneity is the Mokken model: one trait, local independence, and monotonicity – the chance of a higher answer to each item never falls as the trait rises. Under it the sum score orders people by θ, so a higher score can be read as more of the trait without assuming any curve. Monotonicity is checked item by item, by grouping people on their rest score – their sum over the other items – and checking that the share answering at or above each category does not drop from one group to the next; a drop larger than chance is a violation. The café items show none.

Double monotonicity adds one condition to monotone homogeneity: the items’ curves do not cross, so the items keep the same order of difficulty at every level of the trait – an item easier to agree with for the dissatisfied is easier for the delighted too. It is what licenses ordering the items as well as the people. The app checks it two ways: invariant item ordering, item by item, and the rest-score nonintersection check, which looks for a crossing between each pair of items’ step curves. On the café items the nonintersection check finds one small crossing, between quick and slow, and it is not significant: its seriousness index, crit, is 15 and 17 for the two items, where 40 is the bar for a violation worth attention and 80 for a serious one.

Invariant item ordering (IIO) is the property that the items have one order of difficulty that holds for every respondent, not only on average: when it holds, “members agree more readily that the staff were friendly than that the food was good value” is true of each member, and the item order itself can be interpreted – a ladder of skills, a hierarchy of symptoms. It is tested item by item, and summarised by H_T, which is Loevinger’s H computed with the roles of people and items swapped: how sharply the items are ordered, with 0.3 and above supporting an ordering worth using. The café items show no violation and an H_T of 0.026 – nothing contradicts one order, but the six items’ means sit so close together, 3.34 to 3.79, that there is hardly an order to find. Satisfaction items are written to be parallel rather than ranked; IIO is the property to ask of a scale whose items are meant to form a ladder.

Automated item selection – AISP, the automated item selection procedure – partitions a set of items into Mokken scales: it starts a scale from the most scalable pair, adds the item that keeps H highest while every item’s H_i stays at or above the lowerbound c, and when no item can join, starts a second scale from what is left. The choice of c decides how strict a scale is, so the app runs the selection at every c from 0 to 0.55 and a partition is read where it holds across neighbouring values. The café’s six items form one scale up to c = 0.20 and two from 0.25 to 0.40 – taste, fresh and value in one, friendly, quick and slow in the other, the Food and Service of the factor analysis – before value and slow drop out at 0.45; the stable answer is two scales. The search adds one item at a time and never revisits a choice, so it can settle on a good partition rather than the best one.

Reporting #

For an exploratory factor analysis: the items, the correlation type, the extraction method and rotation, the suitability checks (KMO, Bartlett), how the number of factors was decided and what the rules said, the rotated loadings with the communalities and the variance explained, and the factor correlations under an oblique rotation. “Six items; minimum residual extraction with oblimin rotation on Pearson correlations; KMO = 0.73, Bartlett’s χ²(15) = 229.1, p < 0.001; parallel analysis and the Kaiser criterion indicated two factors, which explained 44% of the variance; loadings 0.59–0.76 on Food and 0.58–0.72 in absolute value on Service, the reverse-keyed item loading negatively, no cross-loading above 0.17; r(Food, Service) = 0.39.”

For a confirmatory model: the model in words or a diagram, the estimator, the sample and how missing data were handled, χ² with df and p, CFI, TLI, RMSEA with its 90% interval, SRMR, the standardized loadings or paths with their p-values, and for a scale its AVE and composite reliability. “A two-factor CFA (ML, n = 200) fitted well, χ²(8) = 6.41, p = 0.60, CFI = 1.00, TLI = 1.01, RMSEA = 0.000 (90% CI 0.000–0.071), SRMR = 0.034; standardized loadings 0.57–0.77 in absolute value; r = 0.45; AVE 0.46 and 0.41, CR 0.71 and 0.67.” For invariance, the cascade as a table – each level’s χ², df, CFI and RMSEA with the Δχ² and ΔCFI to the level before – and the level reached.

For an IRT analysis: the model and estimator, the unidimensionality and local-independence evidence, the item parameters with standard errors, the item-fit statistics, the marginal reliability or the test-information range, and any DIF tested with its effect sizes. Say which θ score was used. For a Mokken analysis: H with its standard error and the item H_i, the monotonicity and invariant-item-ordering checks with H_T, and the item-selection partition with the c it was read at.

Where this appears in DataSuite #

• Factor analysis – suitability checks, the number-of-factors rules and their plot, PCA and EFA with every extraction and rotation, loadings, communalities, variance explained, factor correlations and scores, fit indices for an EFA, and bifactor and Schmid–Leiman models.
• Confirmatory factor analysis – the measurement model editor, fit indices, standardized estimates, R², modification indices, residuals, convergent and discriminant validity, and the measurement invariance cascade with latent means.
• Structural equation modeling – regressions among latent variables, path analysis, mediation with bootstrap indirect effects, multi-group models and model comparison.
• IRT analysis – Mokken scale analysis with its scalability coefficients, monotonicity and ordering checks and automated item selection; item parameters under every model family above, exploratory and confirmatory multidimensional models, characteristic and information curves, θ scores by all four methods and their standard errors, reliability and separation, item and person fit, local dependence, DIF and the Wright map.
• Reliability analysis – α, ω and the classical coefficients for a scale whose structure this page’s models confirm; reverse-scoring of items.
• Data transformation – reverse scoring a copy of an item, as a new variable, when a scale score needs one; the answers themselves stay as given, which the IRT module’s careless-responding screen needs.