Regression basics #

A comparison asks whether groups differ; a correlation asks whether two variables move together. A regression asks the next question: by how much does the dependent variable change when a predictor changes, with the other predictors held still – and what would it be for a case we have not seen. Every regression card the app draws, from a straight line through a scatterplot to a logistic model of a yes/no answer, is read with the same dozen ideas, and this page is those ideas. The numbers come from the café data with its three fixes applied, so spend_after has 199 values, age 195 and a model using both runs on 194 members.

From correlation to regression #

Correlation measures how closely two variables move together, on a scale from −1 to +1, without saying which drives which or by how much: the coefficient r is the same whichever variable is called the outcome. Spend before and after the menu change correlate at r = .69 across the 199 members: members who spent more before tend to spend more after. What r does not say is what a member who spent 12 before can be expected to spend after, or whether that expectation changes once the branch is known – and those are the questions a regression answers.

Scatterplot is the picture both start from: one point per case, the predictor along the bottom and the dependent variable up the side. For spend_after against spend_before the cloud rises to the right, tighter at the low end and looser at the top, with one point far below the rest – member 179, who spent 17.41 per visit before the change and 6.35 after. The regression line is the straight line drawn through that cloud so that the points’ vertical distances from it are, in squared total, as small as they can be; the correlation module draws the plot with the line and its band on it.

LOWESS smoother is a curve drawn through a scatterplot without assuming its shape: at each position along the x-axis a short straight line is fitted to the nearest two thirds of the cases, weighted so that the closest count most, and the local fits are strung into a curve; three further passes down-weight whatever the curve has already left far behind, so one or two stray points cannot drag it. It estimates nothing a table reports – no coefficient, no interval, no p – so it is a shape check rather than a result: where the curve runs straight, a straight line and Pearson’s r describe the pair, and where it bends they do not. The correlation module draws it dashed beside Pearson’s line, and on its own for the methods that fit no line, Spearman’s and Kendall’s among them, whenever the pair has at least five distinct x values. For overall against wait (178 members) the line falls at −0.197 points per minute, from 8.4 at no wait to 4.4 at the longest, and r = −.30 is that one slope; the smoother drops steeply over the first minutes – 8.9 at one, 7.8 at four, 6.6 at eight – flattens near 6 past ten minutes, and turns upward at the far end, where the two members who waited longest (16.3 minutes, rated 5, and 19.9, rated 8) are the only cases and the second sits three and a half points above the line. The two of them also tilt the line, whose slope is −0.24 without them: a global fit answers to every point at once, a local one only to its neighbours. What the curve says and r cannot is that satisfaction is lost early in the wait and little further after – a minute is not worth the same at every length. The same curve is the solid line on a regression’s residual plot, where a bend is the first sign that the model’s straight line missed one.

Regression is a model of a dependent variable as a function of one or more predictors: an equation that gives, for any combination of predictor values, the value of the outcome to expect. In the app’s role vocabulary the outcome is the dependent variable and the inputs are the predictors; the fitted equation for the café is spend_after = 3.54 + 0.716 × spend_before. The model does two jobs at once – it describes the relationship (each unit of prior spend is worth 0.716 units of later spend) and it predicts (a member at 12 before is expected at 12.14 after) – and the second job is where it earns its keep. A nurse regresses days to recovery on age and dose; a teacher regresses the exam score on hours of study and the class; a marketer regresses conversions on ad spend and the weekday. The nouns change, the roles do not: one dependent variable, one or more predictors, a line.

Dependent variable is the outcome the model explains – the column whose values the equation produces. It is the y of the scatterplot, the left-hand side of the equation, and the only variable with a residual. A linear regression wants it continuous; a yes/no or a count outcome wants one of the models beyond a straight line.

Predictor is a variable on the right-hand side of the equation: the thing the outcome is modelled from. Other names for the same role are independent variable, explanatory variable and regressor, and a predictor included only so that the others are read with it held constant is a covariate. A predictor can be numeric (spend_before, age) or categorical (branch); the two enter the model differently, and categorical predictors have their own section below.

Linear regression is the model in which the dependent variable is a straight-line function of the predictors plus an error:

“Linear” describes how the coefficients enter – each predictor contributes its coefficient times its value, added up – not the shape of the data, which may be curved and still be fitted by a model with a squared term in it. The app’s regression type Linear is this model, estimated by ordinary least squares.

Multiple regression is a regression with more than one predictor; with a single predictor it is a simple regression. The café’s line of spend_after on spend_before is simple; the same outcome on spend_before, age and branch is multiple, and the difference is not merely more rows in the table – each coefficient now describes its predictor with the others held constant, which is what makes the model able to say whether the branches differ once prior spend is allowed for. The section More than one predictor below is about that.

Covariate is a predictor that is in the model to be adjusted for rather than to be interpreted: a nuisance the outcome depends on, held constant so that the coefficient of interest is read net of it. In the café’s ANCOVA spend_before is the covariate and branch the factor of interest; in a regression the word is a role, not a different kind of column – the app fits every predictor the same way, and which one is “the covariate” is the analyst’s reading of the table.

The line and its coefficients #

Coefficient is a number the model multiplies a predictor by: how much the dependent variable changes, in its own units, when that predictor rises by one unit and every other predictor stays where it is. The coefficient of spend_before is 0.716 – a member who spent one unit more per visit before the change is expected to spend 0.716 more after it. It is the first thing to read on any regression card, before its p-value, because it is the answer to the question the model was fitted for: how much.

Intercept is the value of the dependent variable the model predicts when every predictor is zero – the constant term, printed as (Intercept) at the top of the coefficient table. The café’s intercept is 3.54: the expected after-spend of a member who spent nothing before. No such member exists (the smallest spend_before is 1.50), so the number anchors the line without describing anyone, and its p-value tests whether the line passes through the origin, which is rarely a question worth asking. Centering the predictor at its mean, 8.66, turns the intercept into 9.75 – the mean spend_after, the expected outcome of an average member – which is a number worth reading.

Slope is the coefficient of a numeric predictor seen as the tilt of the line: rise over run, 0.716 units of spend_after per unit of spend_before. Two lines with the same slope run parallel; a slope of zero is a flat line, the model’s way of saying the predictor makes no difference; a negative slope falls to the right, as overall on wait does at −0.197 points of satisfaction per minute waited.

Term is one row of a coefficient or ANOVA table: a predictor, one contrast of a categorical predictor, an interaction, or the intercept. A categorical predictor with k levels contributes k − 1 terms; an interaction contributes one per pair of contrasts. The count of terms, plus one for the intercept, is the p that the degrees of freedom and several thresholds on this page are built from.

B is an unstandardized coefficient – in the raw units of the outcome per raw unit of the predictor, which is the coefficient as defined above. It is the number to report when the units mean something (0.716 units of spend per unit of spend, −0.197 satisfaction points per minute) and the one to use for prediction. The app’s coefficient plot has a Coefficient scale switch between this and β.

β is a standardized coefficient – the change in the outcome in standard deviations per standard deviation of the predictor:

It puts predictors measured in different units on one footing: in the model of overall on taste and wait, B is 0.807 per scale point of taste and −0.129 per minute of waiting, which cannot be compared, and β is .494 and −.195, which can – a standard deviation of taste moves satisfaction two and a half times as far as a standard deviation of waiting. With one predictor β is the correlation: the simple line’s β is .691, which is r. For a categorical predictor the app divides by SD(y) only, since a 0/1 contrast has no meaningful SD, so its β (0.137 for Station) is the category difference in outcome-SD units and is not comparable with a numeric predictor’s; the card says so under the table. In a logistic or count model β is standardized on the predictor alone, because the outcome has no SD on the link scale.

Standard error of a coefficient is the precision of the estimate: how far the coefficient would typically fall from its true value across repeated samples, the standard error of any estimate applied to this one. The slope of 0.716 has SE 0.053, so the 95% CI is 0.716 ± 1.97 × 0.053 = [0.61, 0.82]; the t beside it is the coefficient over its SE, 0.716 ⁄ 0.053 = 13.43 on 197 degrees of freedom (n − p, 199 − 2), and the p is that t referred to the t distribution. The null every coefficient row tests is this coefficient is zero – the predictor adds nothing once the others are in – so p < .001 says the slope is not zero, and the interval says how large it is. Three things shrink the SE: a larger sample, a wider spread of the predictor, and a smaller residual spread; a predictor’s SE also grows with its collinearity with the others.

Fitted values and residuals #

Fitted values are the outcomes the equation gives back for the cases it was fitted to: each member’s predictors put through the line. Member 1 spent 7.85 before, so the model’s fitted value is 3.54 + 0.716 × 7.85 = 9.17 – and the member actually spent 6.37. The fitted values are the line itself, one point per case; their mean equals the mean of the outcome, and their spread is the part of the outcome the model accounts for.

Predicted is the same quantity for any case, seen or new – the value the model assigns from the predictors alone. A member at 8 before is predicted at 9.27 after; at 12, 12.14. In a classification table the word labels the class the model assigned to each case, against the class it actually belonged to.

Residual is what the model missed: the observed value minus the fitted one. Member 1’s residual is 6.37 − 9.17 = −2.80, an after-spend 2.80 below the line; member 179’s is 6.35 − 16.01 = −9.66, the largest in the file. The residuals sum to zero by construction and their spread – SD 2.67 here – is the part of the outcome the model does not account for. Everything the app checks about a linear model it checks on the residuals: whether they are normal, evenly spread, free of pattern and independent, and which cases have too much say. The word also names the last row of an ANOVA table, where it is the sum of squares no term explained.

Residual standard error is the typical size of a residual: the square root of the residual sum of squares over its degrees of freedom, √(1412.9 ⁄ 197) = 2.68 – a little larger than the residuals’ plain SD, since it divides by n − p rather than n − 1. It is in the outcome’s units, so it says how far, in spend, a member’s actual value typically lies from the model’s: about 2.7 per visit either way. The app prints it as Root MSE, the root of the mean squared error, which is the same number.

Standardized residual is a residual divided by its own standard error, so that residuals can be compared to a normal scale: about 5% should lie beyond ±2 and almost none beyond ±3. The simple café line has 9 of 199 beyond ±2 (4.5%) and one beyond ±3 – member 179 at −3.67. The app’s residual Q–Q plot and its count of large residuals use the internally studentized form, which allows for each case’s leverage; the outlier test uses the externally studentized one.

Residual plot is the residuals plotted against the fitted values – the app’s Residuals vs fitted values panel. A well-specified model scatters its residuals in an even, patternless band around zero: a curve in the band means the relationship is not straight (linearity), a funnel means the spread depends on the level (heteroscedasticity), and a point far outside it is a case the model does not fit (member 179, at the bottom of the café’s plot). The LOWESS smoother the app draws through the points makes the first two visible; the plot is the first diagnostic to look at and the tests below it are the numbers behind what it shows.

Prediction interval is the range a single new case is expected to fall in, given its predictors; the confidence interval for the line at that point is the range the mean of such cases is in, and the two differ by the whole residual spread. For a member at 12 before, the line is at 12.14: the 95% interval for the mean after-spend of all such members is [11.63, 12.65], and the 95% prediction interval for one member is [6.83, 17.45]. The narrow band around the line on a scatterplot is the first; the one that would hold most of the points is the second, and it is the honest answer to “what will this member spend”.

Extrapolation is a prediction outside the range of predictor values the line was fitted on – the equation will give a number for a member at 20 before (17.87), but the largest spend_before in the file is 18.94 and only twelve members lie above 15, so nothing in the data says the line stays straight out there. A regression is trusted between the smallest and largest predictor values it saw, and read as a guess beyond them.

How well the line fits #

The first number on the model-fit table is R², the share of the outcome’s variance the model accounts for – .478 for the simple café line, so about half of the member-to-member variation in after-spend is carried by before-spend and half is not. The rows under it are its companions.

Sum of squares is the accounting behind R²: the outcome’s total variation, $\sum {(y-\bar{y})}^2$, split into the part the model reproduces, $\sum {(\hat{y}-\bar{y})}^2$, and the part it leaves in the residuals, $\sum {(y-\hat{y})}^2$. For spend_after on spend_before the total is 2706.0, the model 1293.1 and the residual 1412.9; R² is 1293.1 ⁄ 2706.0 = .478, and the residual sum over its degrees of freedom is the residual variance, 7.17, whose root is the residual standard error. Every F-test on a regression card is a ratio of two such sums, each divided by its degrees of freedom.

SS is a sum of squares as a column header: the variation attributed to the term in that row.

MS is a mean square, the sum of squares divided by its degrees of freedom – a variance. The ratio of a term’s mean square to the residual mean square is its F.

Adjusted R² is R² corrected for the number of predictors. Adding any predictor, even a column of random numbers, raises R² a little, because the line can always use one more degree of freedom to fit the sample’s noise; the adjustment charges each term for that, so adjusted R² rises only when the term earns its place and can fall below zero for a model that explains nothing. On the same 194 members spend_after on spend_before has R² .465 and adjusted .462; adding branch gives .470 and .464; adding age as well gives .470 and .461 – R² went up by five hundred-thousandths and adjusted R² went down, which is the table’s way of saying age is not earning its row. The gap between the two is the shrinkage the app comments on.

Shrinkage, on a model-fit table, is R² minus adjusted R²: how much of the fit is the model’s freedom to chase noise rather than signal. Under .01 the app calls minimal, .01–.05 acceptable and above .05 notable – a sign that the model has more terms than the sample can support and would fit new data worse than it fits this one. The regularized methods on the estimation menu use the same word for a different thing, the deliberate pull of every coefficient toward zero that buys a smaller test error at the price of a biased estimate – the subject of shrinking the coefficients.

f² is R² restated as an effect size, $R^2/(1-R^2)$: the explained variance relative to the unexplained. The three-predictor café model’s R² of .470 is f² = .886; Cohen’s bands for f² are .02, .15 and .35, so it is large by a wide margin, and the app’s Large effect size beside R² is read on them.

F-statistic is the test of the whole model against the intercept-only model: the mean square the terms explain over the residual mean square, with the terms’ degrees of freedom and the residual’s. For the simple line F = 1293.1 ⁄ 7.17 = 180.3 on 1 and 197 df; for the three-predictor model F(3, 190) = 56.1. The null is every coefficient is zero at once, and with one predictor the test is the slope’s t squared (13.43² = 180.3). A significant F says the model beats a flat line, which a large sample makes easy; it does not say the model is good, which is R²’s job, nor which terms carry it, which is the coefficient table’s.

df (regression, residual) are the F-statistic’s two degrees of freedom: the number of terms the model estimates beyond the intercept, and the cases left over, n − p. The three-predictor café model prints 3, 190 – three terms, 194 members minus four coefficients.

p-value (F) is the p-value of the F-statistic, under the null that no predictor has any effect. The app’s Model is significant beside it means only that.

ANOVA – analysis of variance – is the partition of an outcome’s variance into the parts its terms account for and a residual, each part tested with an F. As a test of its own it compares group means, and the comparison module is where its one-way, factorial and repeated-measures forms live; on a regression card it is the table below that tests each term as a whole.

ANOVA table is the regression card’s term-by-term partition: one row per predictor with its sum of squares, degrees of freedom, mean square, F and p, and a residual row. Its use over the coefficient table is for a categorical predictor with three or more levels, which the coefficient table spreads over several contrasts and the ANOVA table tests once. The app’s table uses Type II sums of squares – each term given every other term that does not contain it – so for the three-predictor café model spend_before has SS 1050.50 and F(1, 190) = 144.3, age 0.13 and F = 0.02, branch 11.67 and F = 1.60, p = .207, over a residual of 1383.10 on 190 df (MS 7.28); the η²p column is each term’s share of what the others left, $\text{SS}/(\text{SS}+{\text{SS}}_{\text{residual}})$, the partial η² of the comparison module.

Type of sums of squares is the rule for which other terms each row of an ANOVA table is tested against, and it matters only once the model holds an interaction or correlated predictors. Type I is sequential – each term given only those entered before it, so the table changes with the entry order and its rows add up to the model’s own sum of squares; Type II gives each term every other term that does not contain it, so a main effect is tested with the interactions it appears in set aside; Type III gives each term all the others, interactions included, so a main effect is read at the centre of the terms it interacts with. With no interaction in the model, Types II and III agree – the café rows above are both – and Type I joins them only when the predictors are uncorrelated. The app’s regression card uses Type II unless told otherwise; the comparison module’s factorial ANOVA defaults to Type III.

Model fit is the block of a results card that says how well the model reproduces the data as a whole – on a regression card R², its adjusted form, f², the F-test, the residual standard error and the information criteria; on a logistic or count card the pseudo-R²s, deviances and likelihood-ratio test; on a structural model the fit indices. It is read before the coefficients, because a coefficient in a model that fits nothing describes nothing.

Coefficients is the table with one row per term: the estimate, its standard error, the test statistic and p-value, and the interval, with the standardized column and – for logistic and count models – the exponentiated form beside them. It is where the how much is read, and the model-fit block above it says whether the how much is worth reading.

More than one predictor #

Adjusting for a covariate is what a multiple regression does with every predictor at once: each coefficient is the effect of its predictor with the others held constant, so it answers “does x matter over and above the rest”. The café shows why that changes answers. On the 194 members with an age, Station members spent 1.58 more per visit after the menu change than Riverside members, p = .003 – a branch difference. Put spend_before into the model beside branch and the Station coefficient falls to 0.50, p = .207: Station members also spent more before the change (9.52 against 7.91 per visit), and once each member’s own prior spend is allowed for, the branches’ after-spend differs by half a unit that could well be nothing. The raw comparison was not wrong; it answered a different question – “do Station members spend more” – where the adjusted one answers “does the branch add anything beyond who its members already were”. The app’s ANCOVA is this model with the vocabulary of a comparison.

Confounder is a variable that drives both the predictor and the outcome, so the two move together without either acting on the other. Prior spend is one in the café: Station members spent more before the menu change (9.52 against 7.91 per visit) and more after it, and the branch difference in after-spend – 1.58, p = .003, on its own – falls to 0.50 with spend_before in the model. A confounder you measured is handled by adjusting for it; one you did not cannot be, which is why a regression on observational data estimates an association, and a causal reading needs a design – an experiment, or the stated assumptions a causal estimator rests on, as sequential ignorability is for mediation – rather than a coefficient.

Zero-order correlation is a predictor’s plain correlation with the outcome, ignoring the other predictors – the r of the correlation module. In the model of overall on taste and wait (178 members with all three), taste’s is .534 and wait’s −.298. It is what the outcome and the predictor share including whatever they share through the other predictors.

Partial correlation is the correlation between the outcome and a predictor with the other predictors removed from both – what is left of their relationship once the part each owes to the rest is stripped away. Taste’s partial correlation with satisfaction is .506, wait’s −.226: a little below the zero-order values, because taste and wait are themselves correlated (−.21), so a little of each one’s relationship with satisfaction was carried by the other. It answers “how strongly do these two go together among members alike on everything else”, and its square is the share of the unexplained variance the predictor explains.

Part correlation – the semi-partial – removes the other predictors from the predictor only, and its square is the share of the outcome’s total variance that this predictor explains uniquely: the drop in R² if it were removed. Taste’s is .483 (sr² = .233) and wait’s −.191 (sr² = .036); the model’s R² is .322, so .269 of it belongs to one predictor or the other and .053 is shared between them and cannot be assigned. The correlation module’s partial mode computes both for any pair with any set of controls; the regression card prints them for each term.

The correlations table prints them as Zero-order (r), Partial and Part (sr); the interpretation beside the first reads on the same strength bands as any correlation.

Suppression is the confounder’s mirror image: a third variable that hides a relationship instead of manufacturing one, so that the coefficient of interest grows, appears from nothing or changes sign once the third variable is held constant. A confounder drives the predictor and the outcome the same way and so inflates their raw correlation; a suppressor is tied to the predictor but not to the outcome – or to the two in opposite directions – so it adds to the predictor variation the outcome does not share, and removing it leaves a predictor that tracks the outcome more closely than the raw one did. Turn the café’s branch example around to see the shape: had Station members spent less per visit before the menu change while spending more after it, the raw branch difference would have looked small, and adjusting for spend_before would have enlarged it instead of shrinking it from 1.58 to 0.50. On a card the signature is a partial correlation larger in size than its zero-order value, or of the other sign – which is why the correlation module’s partial mode prints the two beside each other – and a regression coefficient that grows when a covariate enters is the same signal. The café was generated without a suppressor: its adjustments shrink what they touch, the branch difference and taste’s and wait’s partials alike, which is also the usual case – suppression is the exception to recognise when it appears, not to expect. A sign that flips among the weak edges of a full partial-correlation network, where taste–friendly goes from .21 to −.08 once overall is among the controls, is not one: the two items were linked through the disposition they both feed, and holding it fixed removes the link and slightly overshoots.

Multicollinearity is predictors that are correlated with one another, so that the model cannot tell their effects apart. It does not bias the coefficients or hurt the model’s predictions; what it does is inflate the coefficients’ standard errors, because the data hold little information about what happens when one predictor moves and its twin does not. The café’s visits_before and visits_after correlate at .70. Alone, each predicts overall – 0.075 per visit before, p = .003, and 0.039 per visit after, p = .075; together, the before coefficient is 0.086, p = .014, and the after one −0.014, p = .653, with standard errors 1.4 times what they were alone. The pair carries one piece of information – how often the member comes – and the model has handed it to one of them and told the other it adds nothing. The six satisfaction items, correlated at .1 to .5, sit at VIF 1.3 to 1.5 and are untroubled. The limit case is exact collinearity – a food score entered beside the three items it is the sum of – where one term is a pure combination of the others and the app drops it as aliased and says which. The remedies are to drop or combine the twins (a scale score in place of its items – the food and service scores have VIF 1.1 and 1.2), to centre the parents of an interaction, to fit ridge, which shares the twins’ common effect between them, or to accept the wide intervals and read the terms jointly rather than one at a time.

VIF – the variance inflation factor – is how many times larger a coefficient’s variance is than it would be if the predictor were uncorrelated with the others: $1/(1-R_j^2)$, where $R_j^2$ is the R² of that predictor regressed on the rest. The visits pair each have VIF 1.99 – a variance doubled, a standard error √1.99 = 1.41 times wider, which is the inflation seen above. The app flags VIF from 5 as moderate and from 10 as high (thresholds); a categorical predictor with several levels gets one generalized VIF for the whole set, scaled so that it reads on the same bands.

Tolerance is 1 ⁄ VIF – the share of a predictor’s variance that the other predictors do not explain. spend_before’s tolerance of 0.91 beside age and branch says nine tenths of its variation is its own; the visits pair’s 0.50 says half of each is the other.

Collinearity diagnostics is the card that prints VIF and tolerance for each term with a verdict; it is off by default and on the additional statistics panel.

Categorical predictors and interactions #

Categorical predictor is a predictor whose values are groups – branch, with Riverside and Station – rather than amounts. A line cannot be tilted along “Riverside” and “Station”, so the model recodes the groups as numbers first and reports one coefficient for each group but one.

Dummy coding is that recoding: for a predictor with k levels, k − 1 columns of 0 and 1, one per level except the first, each marking “this case is in that level”. branch becomes one column, branch[Station], that is 1 for Station members and 0 for Riverside; a four-level predictor would become three. The coefficient of a dummy is the difference in outcome between its level and the level with no column – so the three-predictor café model’s branch: Station row, 0.50, is how much more a Station member spends after the change than a Riverside member of the same age and prior spend.

Reference category is the level with no column – the one the others are measured against. The app takes the first level in sort order (Riverside), lists it under Reference categories below the coefficient table, and every category coefficient reads as “relative to this”. It is a choice of yardstick, not a finding: with Station as the reference the row would read −0.50 for Riverside and every fitted value and R² would be unchanged. Choose the level that makes the comparisons wanted – a control group, the largest group, the usual condition – by recoding it to sort first.

Interaction is a term for two predictors whose effects depend on each other: the slope of one differs by the level, or the value, of the other. It is entered as their product, and its coefficient is the change in one predictor’s slope per unit of the other. Does prior spend carry over into after-spend the same way at both branches? Fitting spend_after on spend_before, branch and their product gives Riverside a slope of 0.743 and Station 0.675; the interaction coefficient is the difference, −0.068 (SE 0.113), p = .549, and the test of the product term against the additive model is F(1, 195) = 0.36. The slopes do not differ, the additive model stands, and one line with a branch offset describes both. When an interaction is present, the main effects of its parents change meaning: each is the slope of its predictor when the other is zero, which for spend_before in the same model means a member who spent nothing – the reason to centre first.

Moderation is the name for the same idea from the point of view of the question: a moderator is a variable the effect of interest depends on. “Does the taste rating matter less to satisfaction when the wait was long?” is a moderation question, with wait moderating the effect of taste; the model is overall on taste, wait and taste × wait, and the product’s coefficient is −0.056 per minute, p = .091 – suggestive, not established. The app’s moderators role builds the product and adds the simple slopes.

Simple slopes are the effect of a predictor at chosen values of its moderator – the interaction spelled out. Taste’s slope on satisfaction is 0.97 per scale point for a member who waited 2.3 minutes (one SD below the mean wait), 0.80 at the mean of 5.2 minutes and 0.64 at 8.1 (one SD above), each with its own standard error and test; all three are far from zero, so taste matters throughout and the interaction, if real, is a matter of degree. The app plots them as one line per moderator level and, for a continuous moderator, adds the Johnson–Neyman range where the slope is significant.

Johnson–Neyman region of significance is the range of the moderator over which a predictor’s conditional effect is significant – the answer the three probes only sample. Rather than testing the slope at three values, it solves for the moderator values at which the slope’s t equals the critical value and reports the side of them on which the effect holds. For taste’s effect on satisfaction across the wait, the slope is significant for every member who waited less than 11.4 minutes and indistinguishable from zero beyond that: at 11.4 minutes it is 0.46 per scale point (SE 0.23), and at the longest wait in the file, 19.9 minutes, −0.02 (SE 0.50) – too few members waited that long for the data to say what taste does for them.

Johnson–Neyman boundary is one of those crossing values – there can be two, one or none – reported only when it falls inside the observed range of the moderator, since a crossing beyond the data is the fitted line extrapolated, not a finding. The taste-by-wait model’s other solution is at −84 minutes, which no one waited and the table does not print.

Observed moderator range is the span of moderator values the sample holds – 0.3 to 19.9 minutes for wait – printed so that a boundary is read against it, and Observed cases in the significant region is the share of the actual cases whose moderator value falls where the effect is significant, 97% of the 178 members here, so that a region that sounds wide is read for how much of the sample it covers.

Centering is subtracting a predictor’s mean from every value, so that zero means “average” instead of “none”. It changes no slope, no fitted value and no R²; what it changes is the meaning of the intercept and of the main effects in a model with interactions, and the collinearity between a product term and its parents. Uncentered, taste, wait and their product have VIF 3.9, 7.6 and 8.6, because the product of two positive variables tracks each of them; centred, 1.05, 1.11 and 1.07. The app centres the parents of an interaction before it computes VIF for this reason, and says so on the collinearity card; the coefficients it reports are on the original scale unless the predictors were standardized in Data transformation first.

Mediation #

A moderator says when an effect holds; a mediator says how it comes about. The question is whether the effect of a predictor on the dependent variable runs through a third variable – one the predictor moves, and one that moves the outcome in turn – and, if so, how much of it. Does a better-tasting menu make members recommend the café because it makes them more satisfied? The models on this page answer it with three regressions and one product, and the numbers below come from two chains on the café data: taste → overall → spend_after, whose outcome is continuous, and taste → overall → recommend and wait → overall → recommend, whose outcome is a yes or a no.

Mediation is an effect that travels through an intermediate variable: the predictor changes the mediator, and the mediator changes the outcome. In the app’s role vocabulary the chain is predictor → mediator → dependent variable, and the mediators bucket runs the analysis for every predictor × mediator pair. The claim is causal – taste raises satisfaction, which raises recommending – and no amount of fitting makes it one; sequential ignorability says what has to be true for the numbers to carry that meaning.

Mediator is the variable in the middle – the mechanism. It has to be numeric, because the model of it is a linear regression, and it has to come between the predictor and the outcome in time or in logic: a variable measured before the predictor cannot carry its effect. In the café, overall satisfaction is a plausible mediator of taste on recommend and of wait on recommend; branch is not, because nothing a member rates changes which branch they belong to. A variable that changes the size of an effect rather than carrying it is a moderator.

Path a is the predictor’s effect on the mediator – the predictor’s coefficient in a regression of the mediator on it and on any covariates. Regressing overall on taste over the 192 members with an after-spend gives a = 0.87 (SE 0.10, p < .001): a scale point of taste is worth nearly nine tenths of a point of satisfaction.

Path b is the mediator’s effect on the outcome with the predictor held constant – the mediator’s coefficient in a regression of the outcome on both. In the model of spend_after on taste and overall, b = 0.42 (SE 0.17, p = .012): among members who rate the taste alike, each point of satisfaction is worth 0.42 more spend per visit. Holding the predictor constant is what makes b the mediator’s own contribution rather than a restatement of the predictor’s.

Total effect is the predictor’s effect on the outcome with the mediator out of the model – the plain regression of the outcome on the predictor, written c. spend_after on taste alone gives c = 0.50 (SE 0.23, p = .030). It is everything the predictor does to the outcome, through the mediator and otherwise, and the two entries below split it.

Direct effect is the predictor’s effect on the outcome with the mediator held constant – the predictor’s coefficient in the same regression that gives path b, written c′. Beside overall, taste’s coefficient on spend falls from 0.50 to c′ = 0.13 (SE 0.27, p = .628): among members equally satisfied, a point of taste is worth almost nothing more per visit. It is the part of the effect that does not run through the mediator – through some other route, or through nothing the data measured.

Indirect effect is the part that does run through the mediator: path a times path b, the change in the mediator per unit of the predictor times the change in the outcome per unit of the mediator. For taste → satisfaction → spend it is 0.87 × 0.42 = 0.37 per scale point, and for a continuous outcome it is exactly the total effect minus the direct one, 0.50 − 0.13 = 0.37. The card gives it no p-value, because a product of two estimates is not normally distributed even when both factors are; its test is a bootstrap interval, bias-corrected for the product’s skew – 2,000 resamples give [0.11, 0.64], clear of zero, so satisfaction carries a real part of taste’s effect on spend.

Sobel test is the older test of the indirect effect, which treats a × b as if it were normally distributed: its standard error is $\sqrt{a^2\text{SE}{(b)}^2+b^2\text{SE}{(a)}^2}$, the ratio of the product to it is read as a z, and the interval is the product ∓ 1.96 of that error. For taste → satisfaction → spend it gives SE = 0.150, z = 2.44, p = .015 and [0.07, 0.66] – the same verdict as the bootstrap’s [0.11, 0.64], with the lower bound pulled toward zero by the symmetry the test assumes and the product does not have. It is anti-conservative in small samples and wherever the product is skewed, which is why the app’s mediation analysis bootstraps instead and the test appears only on the model comparison card, where a bootstrap for every candidate model would be prohibitive.

Proportion mediated is the indirect effect as a share of the total, $a\times b/c$: 0.37 ⁄ 0.50 = .74, so about three quarters of what taste does to spend it does through satisfaction. The card reports it only when the total effect is itself significant, because a ratio over a near-zero total swings wildly – the wait chain below is the case – and it stops being a proportion when the indirect and direct effects pull opposite ways, where it can exceed 1 or fall below 0; the note under the table says which.

Total indirect effect is, with several mediators in the model, the sum of a predictor’s indirect effects through all of them – its whole mediated effect, which for a continuous outcome equals c − c′. The mediators are fitted in parallel, each path b controlling for the other mediators, so the specific indirect effects are each mediator’s own share and the total is what they add up to; a chain in which one mediator feeds the next is a path model, drawn on the Advanced tab.

Causal mediation is the route the app takes when the outcome is not continuous – a yes or no, an ordered scale, a count. Path a is a linear coefficient and path b a log-odds or log-count one, so their product means nothing; instead the model simulates, for every case, the outcome with the mediator set to the value it takes under each level of the predictor, and averages the differences. The effects come out on the outcome’s own scale – a change in the probability of recommend – with quasi-Bayesian intervals and p-values from the same simulation, under the two names below.

ACME – the average causal mediation effect – is the indirect effect of causal mediation: the average change in the outcome that comes from moving the mediator as the predictor would move it, with the predictor itself held still. Raising taste from 2.3 to 4.7 (the treatment contrast) raises the probability that a member recommends the café by 0.26 through overall satisfaction, 95% CI [0.17, 0.34], p < .001, over the 193 members with an overall score. It is computed once with the predictor held at the low value and once at the high, and the card reports their average; an ordinal outcome gets both, per outcome category.

ADE – the average direct effect – is the direct effect of causal mediation: the change in the outcome from moving the predictor with the mediator held where it was. For taste it is 0.10 [−0.03, 0.24], p = .125: once satisfaction is held still, a better taste rating does little to the recommendation on its own. The total effect is the sum of the two, 0.36 [0.22, 0.50] – a member at 4.7 on taste is 36 points more likely to recommend than one at 2.3 – and the proportion mediated .72.

For wait the same analysis over 178 members reads differently. Moving the wait from 2.2 to 8.1 minutes changes the probability of recommending by −0.15 through satisfaction [−0.23, −0.08], p < .001, and by 0.01 directly [−0.12, 0.13], p = .893; the total effect, −0.14 [−0.28, 0.00], misses the threshold at p = .053, so the card withholds the proportion mediated. The mediated part is established and the whole is not – which is no contradiction: the indirect effect is estimated more precisely than the total, whose interval has the direct effect’s noise in it too. A long wait costs the café recommendations through the satisfaction it costs, and by no other route the data can see.

Treatment contrast is the pair of predictor values a causal mediation effect is stated between, because on a probability or a count scale the effect of one unit depends on where the unit is taken – the same step moves a probability near .5 far more than one near .95. The app takes a numeric predictor from its mean − 1 SD to its mean + 1 SD (taste 2.3 → 4.7, wait 2.2 → 8.1 minutes), a binary one between its two values and a categorical one between its first two levels, and the note under the table names the pair; every ACME, ADE and total effect on the card is per that step, not per unit.

Sequential ignorability is the assumption that turns the arithmetic into a causal claim: given the covariates in the model, nothing unmeasured drives both the predictor and the mediator, both the predictor and the outcome, or both the mediator and the outcome. The first two can be bought by randomizing the predictor; the third cannot – no design assigns members their satisfaction – and no test can check it, so a mediation analysis is a well-argued estimate, never a demonstration. The note under every mediation table states it, and the sensitivity analysis says how much of a violation the finding would survive.

Sensitivity to unmeasured confounding asks how strong an unmeasured cause of both mediator and outcome would have to be to explain the indirect effect away. It re-estimates the ACME while letting the errors of the mediator equation and the outcome equation correlate at a grid of values of ρ, and reports where the effect reaches zero. For taste → satisfaction → spend the indirect effect of 0.37 at ρ = 0 is 0.17 at ρ = 0.1, with an interval [−0.12, 0.45] that already contains zero, and it crosses zero at ρ = 0.2.

ρ is the correlation between the residuals of the mediator equation and the outcome equation – exactly the dependence sequential ignorability assumes away. At ρ = 0 the assumption holds and the ACME is the one the card reports; a ρ of 0.3 means an unmeasured cause of both satisfaction and spend that leaves their residuals 30% correlated.

ρ at which ACME = 0 is the breakdown point: the ρ at which the indirect effect reaches zero, searched in steps of 0.1. Published mediation findings commonly break somewhere between 0.1 and 0.4; a breakdown at 0.5 is robust and one at 0.1 is a warning. The taste → satisfaction → spend path breaks at 0.2 – a modest confounder would suffice – and a report of that path says so beside the interval.

Share of residual variance restates the breakdown point as the product of the shares of unexplained variance an omitted confounder would have to account for in both equations, ρ² at the crossing: .04 here, so a confounder explaining a fifth of what each equation leaves unexplained – 4% as a product – would erase the effect.

Share of total variance is the same breakdown restated against each variable’s total variance rather than its unexplained part: .027 for the café path. Both shares are reported because they are read without knowing what ρ is: a reviewer can ask whether a plausible omitted variable – a member’s mood on the day, say – could explain a few percent of both satisfaction and spend, and the answer here is that it easily could.

Conditional indirect effects are the indirect effect computed at chosen values of a moderator – moderated mediation, the question whether the mechanism itself is stronger for some cases than for others. The app lets the moderator into both paths, so at each probe the effect is a(w) × b(w), and bootstraps them together. Does satisfaction carry taste’s effect on spend less when the wait was long? At waits of 2.3, 5.2 and 8.1 minutes (the mean ∓ 1 SD, over 177 members) the indirect effect is 0.41 [0.06, 0.92], 0.36 [0.12, 0.63] and 0.30 [0.00, 0.69] – a slide from 0.41 to 0.30 whose intervals overlap almost entirely.

Index of moderated mediation is the slope of the indirect effect on the moderator: how much the indirect effect changes per unit of the moderator, one number that says whether the mediation is moderated at all, tested by whether its bootstrap interval excludes zero. When one path carries the moderator it is that path’s interaction coefficient times the other path – taste × wait on satisfaction is −0.055 per minute, so the index is −0.055 × 0.39 = −0.021 per minute of waiting, [−0.07, 0.01]. When both paths carry it, as the app’s table allows, the indirect effect is a curve rather than a line and the row reads Change in indirect effect per unit of the moderator instead: −0.020 per minute here, [−0.12, 0.08]. Either way the interval holds zero, and the mechanism is not shown to depend on the wait.

Comparing models #

Nested models are a pair in which one is the other with terms removed – spend_after on spend_before is nested in spend_after on spend_before and branch. The larger model always fits at least as well; the question is whether the extra terms earn their keep, and for two nested linear models the test is an F on the drop in residual sum of squares: on the 194 members with an age, adding branch to spend_before gives F(1, 191) = 1.65, p = .201, and adding age to both F(1, 190) = 0.02, p = .894 – the same verdicts the coefficient table’s t-tests gave, because a one-term addition is the same test. For generalized linear models the analogue is the likelihood-ratio test on the drop in deviance.

Log-likelihood is how probable the observed data are under the fitted model, on a log scale; higher (closer to zero) is better, and every model comparison on a regression card is built from it. The café’s models of spend_after on the 194 complete members have log-likelihoods of −527.3 with no predictor, −466.6 with spend_before, −465.8 with branch added and −465.8 again with age – the last two additions barely move it. On its own the number means nothing; the differences between models do.

Likelihood-ratio test compares two nested models by twice the difference in their log-likelihoods, which under the null that the extra terms are zero follows a χ² distribution on as many degrees of freedom as terms were added. A logistic card’s χ² (LR test) row is this test of the whole model against the intercept-only one – 65.8 on 1 df for recommend on overall – the generalized model’s F-statistic.

AIC – Akaike’s information criterion – is a score for choosing among models fitted to the same cases: −2 × log-likelihood + 2k, where k is the number of parameters, so that a better fit lowers it and every extra parameter raises it by 2. Lower is better, the number itself is arbitrary, and only differences between models on the same data mean anything. The four café models score 1058.7, 939.3, 939.6 and 941.6: spend_before alone beats the intercept-only model by 119 points, adding branch costs 0.3 and adding age two more – the fit did not improve enough to pay for the parameter. The working rules: a difference under 2 is no difference, 4 to 7 is considerable, above 10 is decisive; and unlike an F-test, AIC can compare models that are not nested – spend_before against visits_before as the sole predictor (939.3 against 1038.2) – which is what the model comparison card is for.

BIC – the Bayesian information criterion – is AIC with a heavier price per parameter, log(n) instead of 2, which for 194 members is 5.3. It prefers smaller models than AIC does and, as the sample grows, points to the true model where AIC keeps a term or two too many; the two agree on the café (949.1, 952.7, 957.9 for the three models). Report whichever the field expects, and read a disagreement between them as the sign of a term the data cannot decide about.

AICc is AIC with a correction for small samples,

where k counts every parameter the model estimated – the intercept and the residual variance included, so a linear model with three predictors has k = 5 – and n is the number of cases every candidate was fitted to. The correction is the part of the price of a parameter that AIC leaves out when the sample is small relative to the model: at 194 members it adds 0.13 to the one-predictor model of spend_after and 0.32 to the three-predictor one and changes no verdict, while at 20 members the same two models would pay 1.5 and 4.3, enough to reverse a close call; as k approaches n − 1 it grows without limit, which is the criterion saying the model has too many parameters for the data. It converges on AIC as the sample grows, so there is no sample on which it is the worse choice, and the app’s model comparison card ranks by it – ΔAICc, each model’s gap to the best, is the number the working rules for AIC differences are read on.

Akaike weight is a model’s AICc difference from the best turned into a share: $\exp (-\mathrm{\Delta }/2)$ for each model, divided by the sum over all the models compared, so that the weights add to 1 and read as the probability that each is the best of the set. The model comparison card fits every subset of the chosen predictors and weights them this way: of the eight models spend_before, age and branch make on spend_after over the 194 members, spend_before alone takes .41, spend_before with branch .33, with age .15 and all three .12, and the four models without spend_before, 110 or more AICc points behind, share less than one part in 10²⁴ between them. A weight is a share of this set – add a strong candidate and every weight falls – and it is the currency of everything below: the evidence ratio, the confidence set, the averaged coefficients and the importance of each predictor.

Evidence ratio is the best model’s weight divided by another model’s – how many times more likely the best model is to be the best than this one – which for a gap of Δ is $\exp (\mathrm{\Delta }/2)$: 2.7 at Δ = 2, 7.4 at Δ = 4, 148 at Δ = 10, so the working rules for AIC differences are these ratios in another dress. In the café’s ranking spend_before alone is 1.2 times as likely as spend_before with branch – nothing to choose between them – 2.8 times as likely as spend_before with age, 3.5 times as likely as the model with all three, and 8 × 10²³ times as likely as the best model without spend_before. It compares two models and nothing else, so it does not move when a third candidate enters or leaves the set, which a weight does; a ratio under about 3 says the data cannot tell the two apart, and one in the thousands says the second model can be forgotten.

Confidence set is the models that cannot be ruled out: the smallest group, taken from the top of the ranking, whose Akaike weights add up to the confidence level – 95% unless the settings say otherwise – read off the Cumulative weight column, which runs down the ranking adding each weight to the ones above it, so that the model carrying the total past the level is the last one in. It is read like a confidence interval for “which model is best”: the set holds the best model with the stated probability, and every model outside it can be set aside. For the café’s eight the running total goes .41, .74, .88 and then 1.00, so the set is the four models that contain spend_before and the four that do not are out. A set of one is a decisive comparison; a set holding most of the candidates is the data saying they are all about as good, which is when the averaged coefficients below, rather than any one model’s, are the answer.

Model averaging is reading the coefficients across the whole set of models rather than from the winner alone: each coefficient is averaged over the candidates, weighted by their Akaike weights, so a term’s estimate comes mostly from the models the data favour and the uncertainty about which model is right becomes part of its standard error. Picking the best model and reporting its coefficients as if no search had happened understates that uncertainty – the second model was nearly as good and says something slightly different – and the averaged table is the repair. The app’s model-averaged coefficients table gives each term two averages, the full and the conditional, and their standard errors add the spread of the estimate between models to the error within each, so an averaged interval carries the model-selection uncertainty that a single model’s interval does not.

Full average is a term’s coefficient averaged over every model in the set, counting it as zero in the models that leave the term out. It is shrunk toward zero by exactly the weight of those models: branch: Station is 0.51 in the two models that contain it, those models hold .45 of the weight, and its full average is .45 × 0.51 = 0.23, with an interval of [−0.44, 0.90] that reads as “the branch difference, allowing that the branch may not belong in the model at all”. A term every good model contains is not shrunk – spend_before’s full average is 0.70 [0.59, 0.81], the same as its conditional one – and a term the data reject is shrunk almost away, age to 0.001 per year. It is the number to report as a predictor’s effect when the question includes whether the predictor matters.

Conditional average is the same coefficient averaged over only the models that contain the term – branch: Station is 0.51 [−0.27, 1.29], close to what either model containing it reports, since the two agree – and answers “how large is the effect, given that the term belongs in the model”. It is the further of the two averages from zero whenever the full one was shrunk, and the right one to report when the term’s place in the model was settled on other grounds; reported on its own for a term of low importance, it overstates what the data support, because it hides that most of the weight went to models without the term. Which to report is the question being asked: the full average says what a predictor is worth across the uncertainty about the model, the conditional what it is worth in the models that use it.

Variable importance is the sum of the Akaike weights of the models that contain a predictor – the share of the evidence that sits on models using it. spend_before scores 1.00, since every model with any weight has it, branch .45 and age .26; the app’s variable importance table ranks the predictors by it and counts, for each, how many models it was in and how many of those gave it a positive or a negative coefficient. It is neither a p-value nor an effect size: a predictor in half of the candidates starts at .50 before the data have said anything (prior inclusion share), and two predictors that carry the same information split the weight between them – visits_after scores .30 beside visits_before’s .94, below its own starting share, because the pair carry one piece of information, how often a member comes, and the weight goes to the one that carries it better (multicollinearity). Read a low score as “not consistently chosen”, not as “not relevant”.

Prior inclusion share is the fraction of the candidate models that contain a predictor before any of them is fitted – the importance it would score if every model weighed the same. In an all-subsets search every predictor is in exactly half of the models, so each starts at .50; a minimum or maximum number of predictors moves it, up to 1.00 for a term the search window puts in every model, whose importance is then 1.00 whatever the data say. Only the distance above the share is evidence for the predictor and only the distance below it evidence against: spend_before’s 1.00 is the whole distance from .50 to 1, branch’s .45 and age’s .26 fall short of their .50, and the app’s importance chart draws each predictor’s share as a tick on its bar for that reason.

Overfitting is a model that has learned the sample’s noise as if it were signal: it fits these cases better than a smaller model does and predicts new ones worse. Its symptoms are an adjusted R² well below R², coefficients that swing when a few cases are dropped, and a cross-validated error larger than the in-sample one. The cures are fewer terms, more cases, or a regularized fit that shrinks the coefficients on purpose.

Cross-validation estimates how a model will do on data it was not fitted to, by splitting the sample into folds, fitting on all but one, scoring on the one left out, and rotating. The app uses it to choose the penalty of a regularized model and to report a cross-validated AUC for a classifier; a model whose cross-validated fit is far below its in-sample fit is overfitting, whatever its R² says.

Checking a regression #

The four assumptions behind a linear model’s p-values and intervals are made about the residuals, and the assumptions page explains each: linearity (the relationship is straight), independence (one case’s residual says nothing about another’s), homoscedasticity (the residuals are as spread out at one end of the line as at the other) and normality of the residuals – not of the outcome, which can be as skewed as it likes so long as what is left after the model is not. The app’s residual diagnostics card runs one test for each of the last three; the goodness of fit card tests the first; and the influence statistics card asks a fifth question the assumptions do not – whether any one case is holding the line where it is.

Normality of residuals is checked with a Shapiro–Wilk test on the residuals and a Q–Q plot of the standardized ones. The three-predictor café model gives W = .991, p = .257 – residuals as normal as could be asked – while overall on taste and wait gives W = .975, p = .002, because the 0–10 satisfaction scale has a ceiling the residuals pile up against. The assumption matters least of the four: with 178 members the central limit theorem protects the coefficients’ tests, and what the failure signals is a bounded or skewed outcome that a different model might describe better.

Durbin–Watson test checks the residuals for autocorrelation: whether each residual resembles the one in the row before it. The statistic runs from 0 to 4 and sits at 2 when neighbouring residuals are unrelated; below about 1.5 they are positively correlated, above 2.5 negatively. The café model’s 1.92 is unremarkable – and meaningless, because the rows are members in the order the loyalty forms were filed, not a sequence. The test is for data where row order is time or position; on a survey it reports the incidental order of the file, and the app says so under the table. Where it does apply, the remedy is a model that knows about the sequence – the Newey–West standard errors or a time-series model.

Breusch–Pagan test checks for heteroscedasticity: whether the squared residuals grow or shrink with the predictors. The simple café line gives BP = 3.69, p = .055, on the edge; the three-predictor model BP = 10.7 on 3 df, p = .013 – the spread of after-spend around the line is not constant, wider for members who spent more before. The coefficients are unaffected; their standard errors are, and the fix is robust standard errors rather than a different model.

Robust standard errors are standard errors computed without assuming the residual spread is constant – the heteroscedasticity-consistent HC3 form – or, for ordered data, without assuming the residuals are independent either (Newey–West). They change the SE, t, p and interval columns and leave the coefficients as they were: the model’s slope is right, its stated precision was not. The app’s standard errors selector switches them on; for overall on taste the slope stays 0.878 and its SE goes from 0.097 to 0.108 under HC3. They are the routine choice whenever the Breusch–Pagan test fails and cost nothing when it does not.

RESET test – Ramsey’s regression specification error test – checks linearity by adding the squared and cubed fitted values to the model and testing whether they explain anything the straight line missed; a significant F says the relationship bends, or a predictor is missing, and the app reports it on the goodness-of-fit card. The three-predictor café model gives F(2, 188) = 0.99, p = .373 – no evidence of misspecification; spend_before on age, which bends because the members come in two age groups, gives p = .003. It is the one diagnostic whose failure means the coefficients themselves, not just their precision, are wrong.

Cases that pull the line – a large residual, a far-out predictor value, or both – are the subject of the outliers page: leverage, Cook’s distance, DFFITS and the outlier test on the studentized residuals, all on the influence card – and the next section holds the models that down-weight such a case instead of deleting it.

Resisting outliers #

Every regression card on this page so far has drawn a least-squares line, and a least-squares line has one weakness the outliers page measures case by case: a single row far from the rest can move it. This section leaves the café’s catering order in – member 136, an after-spend of 66.53 against a before-spend of 18.53, the one value the range recode blanks everywhere else on this page – so that the two families the app offers in place of least squares can be seen pulling away from it, on all 200 members.

Least squares is the rule that places the ordinary regression line: the intercept and slope that make the sum of the squared residuals as small as it can be. Squaring is what makes the line easy to compute and its tests exact when the errors are normal, and it is also what makes one case so powerful. Member 136’s residual, once the line has done what it can for the case, is 47.63, and its square, 2,269, is 60% of the whole residual sum of squares from one row in 200 – so the line cuts that sum faster by moving toward the one case than by fitting the other 199 better, and it does: with the catering order in, spend_after on spend_before is 2.17 + 0.903 × spend_before, R² = .36, residual standard error 4.37; with it blanked, 3.54 + 0.716 × spend_before, R² = .48 and 2.68. One row has moved the slope by a quarter of itself, raised the residual standard error by about 60% (2.68 → 4.37), and lifted the line clear of the crowd – 112 of the 200 members, 56%, lie below it. The influence measures on the outliers page find such a case after the fact; the two families below refuse to let it have that much say in the first place.

M-estimation replaces the squared residual with a loss that grows more slowly once a residual is large, so that a case’s pull on the line is capped. Huber’s, the classic, is quadratic up to a cut-off and linear beyond it – least squares for the cases the line fits, least absolute deviations for the ones it does not – and the fit is found by weighted least squares repeated until it settles, every case past the cut-off weighted by the cut-off over its residual. On the café’s 200 members Huber’s M-estimate is 3.16 + 0.758 × spend_before: member 136 enters at a weight of 0.07 – seven per cent of a case – 36 members are down-weighted at all, and the slope comes back from 0.903 most of the way to the clean line’s 0.716. What it has not done is remove the case: the cap bounds its pull but never sets it to zero, and a case far out along the predictor – high leverage – keeps its say in full, because an M-estimator looks only at residuals. The next step fixes both.

MM-estimation is the two-stage fit behind the app’s Robust linear regression type (robustbase::lmrob). A first, S-estimate finds the line that makes a robust measure of the residual spread as small as possible – a fit with a breakdown point of 50%, meaning that up to half the cases could be moved anywhere without carrying the line with them, where least squares breaks at a single case. Its scale is then held fixed and an M-estimate is run from that start with a redescending weight function, Tukey’s bisquare, which unlike Huber’s falls to exactly zero for a residual beyond a cut-off: 4.685 times the robust scale, 4.685 × 2.65 = 12.4 units of spend here. The result keeps the S-estimate’s breakdown point and regains the efficiency of least squares on clean data – 95% of it when the errors are normal. On the café’s 200 members the MM line is 3.23 + 0.748 × spend_before (SE 0.060), t = 12.48; refitted on the 199 with the order blanked it is 3.22 + 0.748 – the same slope to three decimals and an intercept a hundredth apart, which is the point: the estimator did on its own what the range recode did by hand. The fit summary reads differently from a least-squares one. Robust R², .50, is the same ratio taken on the case weights rather than the raw sums of squares, and not comparable with the OLS .48; a Robust residual scale of 2.65 stands where the residual standard error stood; and a χ² Wald test of the slopes replaces the F, since the estimator maximizes no likelihood and has no AIC or BIC to be compared by.

Case weight is the share of a case that reached the fit, between 0 and 1 – the bisquare weight the M-step converged on, ${(1-{(r/\text{cut-off})}^2)}^2$ for a residual r inside the cut-off and 0 beyond it. It is the robust fit’s influence report, printed in place of Cook’s distance when influence statistics are ticked: a case that agrees with the line keeps nearly all of its weight, one that disagrees mildly loses some, and one that disagrees wildly is rejected – counted for nothing – with the rows named at each step. For the café, the mean weight is .91 and the quartiles [.89, .96, .99], so the bulk of the sample entered at close to full weight; the smallest weight is 0.00, member 136’s, whose residual against the MM line is 49.4 – four times the cut-off – and who is the one member rejected; four members count for less than half a case, member 179 among them at a weight of 0.13 for a residual of −9.9, the outcome outlier the studentized residual test flags once the order is gone. A rejected row is not deleted: it stays in the table, its weight says what the estimator made of it, and the report can quote the number instead of defending a deletion. Two alarms guard the reading – a mean weight below .5, or more than half the rows rejected – and either means the fit describes a minority of the sample rather than its bulk.

Quantile regression fits a line through a chosen quantile of the outcome instead of its mean: the median line has half the cases above it and half below, the 0.9 line nine tenths below, and each is a straight-line function of the predictors with its own intercept and slope. The app’s Quantile regression type (quantreg::rq) fits a grid of them at once – the quartiles by default – and prints a coefficient matrix with the least-squares estimate as its first column, so that a predictor whose slope changes from one quantile to the next is read across its row. Two things it does that a mean line cannot: it resists an outlying outcome, since a case counts by which side of the line it is on and not by how far; and it shows whether a predictor acts differently at the top of the outcome’s distribution than at the bottom – a spread that a single mean slope reports as one number and hides. On the café the second is a modest fan. Across the quartiles-and-tails grid the slope of after-spend on before-spend is 0.67 (SE 0.07) at the 10th percentile, 0.81 (0.07) at the median and 0.77 (0.11) at the 90th, and the intercepts run from 0.72 to 6.88; for a member at 5 before, the 10th percentile of after-spend is 4.05 and the 90th 10.71, for one at 12, 8.71 and 16.07 – the band widens a little as prior spend rises, which is the heteroscedasticity the Breusch–Pagan test reported, and no two slopes differ by more than their standard errors can tell apart, so prior spend shifts the whole distribution of after-spend rather than stretching one end of it.

τ (tau) is the quantile a line is fitted for, between 0 and 1: τ = 0.5 is the median, 0.25 the lower quartile, 0.9 the 90th percentile. The line at τ is found by weighting each residual τ on one side and 1 − τ on the other and making the total as small as possible, and a coefficient at τ is the change in that quantile of the outcome per unit of the predictor, not the change in its mean. Every block of a quantile card says which τ it was computed at: the Quantiles (τ) control picks the grid, and the Reported quantile picks the one τ the diagnostics, the fit summary and the standardized coefficients are read at. That a line is the τ line is easy to check – on the café’s 200 members the shares below the 0.1, 0.25, 0.5, 0.75 and 0.9 lines are .095, .245, .495, .745 and .900.

Median regression is quantile regression at τ = 0.5, and the plainest way to see what a quantile line resists that a mean line cannot. The mean line minimizes squared residuals, so how far a case lies from it matters, and more the further it is; the median line minimizes absolute residuals, so all that matters is which side a case is on. Member 136 is one case above the line whether the order was 66.53 or 666.53, and the median line, 2.62 + 0.805 × spend_before on the 200 members, is exactly the same on the 199 with the order blanked; so are the 0.1 and 0.25 lines, while the 0.75 and 0.9 lines shift slightly (slopes 0.738 to 0.719 and 0.766 to 0.721), since removing a case above them changes which case sits at the quantile. The fit summary’s R¹, Koenker & Machado’s measure – .26 at the median here – is R²’s idea with the quantile loss in place of the squared one, and is comparable neither across τ nor with an R². Against the robust family the median line is the less efficient estimate of the centre – its slope’s standard error is 0.072 to the MM fit’s 0.060 – which is why a median-only fit is largely redundant with it; what quantile regression alone can do is fit the other quantiles beside it.

Shrinking the coefficients #

Every fit so far has been free: the coefficients are whatever makes the residuals smallest, and nothing pulls them back. The regularized estimation methods add such a pull – a penalty on the size of the coefficients – and this section is about what that buys, what it costs and how the app decides how hard to pull. The instance is overall satisfaction on the six satisfaction items – taste, fresh, value, friendly, quick and slow – over the 193 members with an overall score, the predictor set multicollinearity found untroubled at VIF 1.3 to 1.5. Its unpenalized fit is 1.41 + 0.411 taste + 0.342 fresh + 0.202 value + 0.487 friendly + 0.315 quick − 0.159 slow, R² = .61, with every term significant and value (p = .015) and slow (p = .041) the least so.

Regularization is fitting with a penalty added to the loss: the residual sum of squares plus λ times a measure of the coefficients’ size, minimized together, so that a coefficient stays large only when the fit it buys outweighs the penalty it costs. Every coefficient is pulled toward zero – shrunk – by an amount the penalty’s shape and λ set, and the intercept is left alone. Because the pull is on size, the predictors are standardized before the fit so that the penalty falls on each equally whatever its units, and the coefficients are returned on the original scale; a covariate is exempt, forced into the model at every λ. The app’s estimation method menu holds the three penalties below and the grouped form of one of them; the classic fit is the case λ = 0.

Bias–variance trade-off is why a deliberately wrong coefficient can predict better than the right one. An unpenalized estimate is unbiased – right on average across samples – but each sample pulls it a different way, and with many predictors, correlated predictors or few cases per term that spread is large: the coefficients chase the sample’s noise along with its signal, which is the overfitting of an earlier section. Shrinking biases every coefficient toward zero and cuts the spread, and prediction error, which is the sum of the two, falls as long as the variance removed outweighs the bias added. On the café’s six items the trade barely pays: the 10-fold cross-validated error of the unpenalized fit is 1.49 and the best any penalty finds is 1.49, because six well-separated predictors on 193 members were not overfitting to begin with. It pays where it is needed – the visits pair at VIF 2, a model with nearly as many terms as cases – and costs little where it is not, which is the case for reaching for it before a fit has gone wrong rather than after.

Ridge – the L2 penalty – charges λ times the sum of the squared coefficients. It shrinks every coefficient by a proportion that grows with λ and with the predictor’s collinearity, and never sets one to exactly zero, so every predictor stays in the model; what it does for correlated predictors is share their common effect between them instead of handing it to one and denying the other. At the café’s λ.1se of 1.56 the six item coefficients keep between 64% (friendly) and 86% (value) of their unpenalized size – and slow keeps 101%, since under collinearity ridge can move a coefficient either way – for an R² of .57 against .61. The app’s Ridge (L2) card prints that retention as its Shrinkage column; on the model-fit table the same word means something else.

LASSO – the L1 penalty – charges λ times the sum of the absolute coefficients, and the change of shape has a large consequence: a coefficient the penalty outweighs is set to exactly zero rather than made small. The lasso therefore selects – the terms left non-zero are the model – and the card’s Status column reads Selected or Excluded (shrunk to 0) for each term. On the café’s six items at λ.1se = 0.215 the answer is that all six survive: taste 0.360, fresh 0.283, value 0.089, friendly 0.411, quick 0.242 and slow −0.076 – the first four at 77–88% of their unpenalized size, value and slow at 44% and 48%, each a step from zero. Walking λ up from there, the coefficient trace shows the order in which the lasso would let them go – value and slow the last in, at λ = 0.38 and 0.41, taste and friendly the first, at 0.95 – which is a ranking a coefficient table with no standard errors can still give.

Elastic net blends the two: a share α of the penalty is L1 and 1 − α is L2, so that it selects like the lasso and shares like ridge – its case is predictors that come in correlated groups, where the lasso alone picks one of the group more or less at random and drops the rest, and the elastic net brings them in together. The app’s Elastic net mixing (α) slider sets the share, 0.5 by default; on the café’s six items at α = 0.5 the coefficients sit between the two fits (value 0.108, slow −0.096).

Group lasso applies the lasso’s penalty to a predictor’s design columns as one block, so that a categorical predictor with k levels – k − 1 dummies – enters or leaves the model whole instead of one level at a time. It is the penalty for the question “is branch in the model?” rather than “is Station in the model?”; with no categorical predictor every group holds a single column and the fit is the lasso. The café has no predictor with three or more levels to show it on – branch is one column either way, so its group verdict and its lasso verdict are the same – where a nurse’s ward, a teacher’s class or a marketer’s channel would be the typical group.

λ (lambda) is the strength of the penalty – how hard the coefficients are pulled toward zero. At λ = 0 the fit is the unpenalized one; as λ grows the coefficients shrink, and under the lasso drop out one by one, until at a large enough λ only the intercept is left. It is chosen not by the analyst but by the data: the app fits the model along a path of λ values – 56 of them, from 1.05 down to 0.0063, for the café’s lasso – and scores each by cross-validation, the error its model makes on held-out folds. λ.min is the λ with the smallest cross-validated error, λ.1se the largest λ whose error is within one standard error of that minimum, and the path’s ends matter: a λ.min at the smallest λ of the path, as the café’s is (0.0063, CV error 1.49 ± 0.14), means the curve was still falling when the grid ran out and the best fit is the unpenalized one, and the card says so.

1-SE rule is the choice of λ.1se over λ.min, and the app’s default: take the most penalized model whose cross-validated error is not distinguishably worse than the best. The two errors differ by less than the noise in either – the café’s 1.61 ± 0.13 against 1.49 ± 0.14 – so the rule gives away no measurable accuracy and buys a simpler, more stable model, which is what a penalty is usually reached for: not the best predictor but the shortest list of predictors that predicts about as well. λ.min is the choice when prediction alone is the goal.

Selection stability asks how much of a lasso’s selection is the data and how much is the sample: the model is refitted on B random halves of the sample at the λ the full fit selected, and each term’s selection frequency – the share of refits in which it came back non-zero – is reported beside its coefficient. It stands in for the standard error a penalized coefficient does not have, without being one: a term at 100% is selected however the sample is halved, a term at 60% survives on the strength of the rows that happened to be drawn. On the café’s six items 50 half-samples of 96 members give taste, fresh, friendly and quick 100%, value 82% and slow 70% – the two coefficients the lasso had shrunk by half are also the two the halves disagree about.

Error control by complementary pairs is the stability run turned into a procedure with a guarantee. Each of the B draws splits the sample into a half and its complement and refits both, counting a term as selected in a half when it is non-zero anywhere on the λ path down to the selected λ, and from the average number of terms a half selects it bounds the expected number of false selections among the terms whose frequency reaches a threshold π (0.75 by default). The bound is informative only when the halves select a minority of the candidates: on the café’s six items an average half selects 5.6 of them, so a term clears 0.75 on chance alone, and the card withdraws the bound and names the smallest π that would carry one, 0.94. That is the honest reading of a small, well-behaved predictor set – there is nothing to select against – and the bound earns its keep where there are dozens of candidates and a handful of true ones.

Post-selection inference is the route to a p-value after a selection. The obvious one – refit the selected terms without the penalty and read their p-values – is invalid, because the rows that chose the terms now test them, and a term chosen for looking good looks good again. Multi-sample splitting breaks the circle: the sample is split in half, the lasso selects on one half with λ re-tuned there, the selected terms are refitted unpenalized on the other half, and their p-values, Bonferroni-corrected by the number selected, are read off rows that never saw the selection; over B such splits the B values per term are combined by a quantile rule that controls the family-wise error over every term. A term the lasso selects in only some of the splits carries p = 1 from the rest and comes out near 1, which is the procedure working. On the café, 50 splits of 96 and 97 members give taste p = .007 and friendly p < .001 – established after selection – fresh .076 and quick .069, and value and slow 1.0 though the unpenalized fit had them at .015 and .041: they were selected in 82% and 68% of the splits and significant on the other half in too few of those to survive. The card’s Unpenalized estimate beside the p is a different quantity – the terms the full fit selected, refitted once on the whole sample, the size the model implies once the pull is off (value 0.202, slow −0.159) – and carries no test of its own, since that would be the invalid one.

Beyond a straight line #

A straight line is the wrong shape for several common outcomes. A yes/no answer takes only two values, so a line through it predicts probabilities below zero and above one – fitted to recommend (yes = 1) on overall, the line reaches −0.31 at a score of 0 and 1.04 at 10. A rating with a handful of ordered levels, or a choice among several unordered ones, is a category rather than an amount, and its levels have no distances between them for a line to use. A count – visits, complaints, purchases – is never negative, is skewed, and spreads out as it grows. All of them are handled by models that keep the linear equation but put it behind a function that maps it onto the outcome’s own scale; the ordinal, multinomial and two-part count models further down are the same idea with more than one cut or more than one equation, separation and events per parameter are the two ways such a model runs out of data, and how a yes/no model’s probabilities become decisions, and how well, is the section after this one.

Generalized linear model – GLM – is that family: a linear predictor $b_0+b_1{x}_1+\dots$, a link function that connects it to the mean of the outcome, and a distribution for the outcome around that mean (binomial for yes/no, Poisson or negative binomial for counts, normal for the straight line, which is the family’s simplest member). The coefficients are read on the link’s scale and, exponentiated, as multiplicative effects on the outcome’s; the app’s regression type menu is a list of these models.

Link function is the transformation the linear predictor passes through to become the outcome’s mean: the logit for a probability, so that any value of the line lands between 0 and 1; the log for a count, so that it lands above zero; the identity for a continuous outcome.

Linear predictor is the value of the equation before the link is undone – the model’s output on its own scale, log-odds for a logistic model and log-count for a Poisson one. For recommend on overall the linear predictor is −5.16 + 0.848 × overall: −1.77 at a score of 4, −0.07 at 6, 1.63 at 8 and 3.32 at 10. The app plots residuals against it where a linear card plots them against the fitted values.

Logistic regression models a two-level outcome by the log-odds of one level – the modelled outcome, which the card names – as a linear function of the predictors. For the 193 members with an overall score, the log-odds of recommending the café rise by 0.848 per scale point; undone through the logit, that is a probability of recommending of .15 at a score of 4, .48 at 6, .84 at 8 and .97 at 10 – the S-shaped curve a line cannot draw. The coefficient table’s z replaces the linear model’s t, and the model-fit block replaces R² with the pseudo-R²s, the deviances and the likelihood-ratio test. The app’s Binomial logistic type is this model; ordinal and multinomial logistic extend it to outcomes with several ordered or unordered levels.

Log-odds – the logit – is the scale a logistic coefficient lives on: the natural log of $p/(1-p)$, zero at a probability of one half, unbounded either side. A coefficient of 0.848 on this scale is hard to picture, which is why the card also prints its exponential.

Odds ratio is the exponentiated logistic coefficient, e^b, and the way a logistic effect is usually reported: the factor by which the odds of the outcome multiply per unit of the predictor. e^0.848 = 2.34 [1.79, 3.04]: each extra point of overall satisfaction multiplies the odds of a recommendation by 2.3. An odds ratio of 1 is no effect (its interval crossing 1 is the same as the coefficient’s crossing 0), 0.5 halves the odds, 2 doubles them; the effect-sizes page has the same measure for a 2 × 2 table, where it is the same number a logistic model with one binary predictor would print. The card’s OR and OR 95% CI columns are this; the intercept’s odds ratio (0.006) is the odds of recommending at a score of zero and is read as rarely as a linear intercept.

Pseudo-R² is any of several substitutes for R² in a model that has no residual variance to partition: each compares the fitted model’s log-likelihood with the intercept-only model’s, and none reads on R²’s scale. The app prints three, and the model-fit interpretation is on McFadden’s bands.

McFadden’s R² is $1-\ell /{\ell }_0$, with ℓ the model’s log-likelihood and ℓ₀ the intercept-only model’s: .271 for recommend on overall, from −88.3 against −121.2. Its values run far below an OLS R² – McFadden’s own reading calls .2 to .4 an excellent fit – which is why the app’s bands top out at .2 and never award more than Excellent fit. The adjusted form beside it charges for the parameters, as adjusted R² does.

Cox & Snell R² is $1-\exp (2({\ell }_0-\ell )/n)$, .289 here; it cannot reach 1 even for a perfect model, which is the defect the next measure corrects.

Nagelkerke’s R² is Cox & Snell’s divided by its own maximum, so that it runs from 0 to 1: .404 for the café’s model. It is the largest of the three and the one most often reported as if it were an R²; it is not one, and the three should be read as ranks – a higher value is a better fit – rather than as shares of variance.

Deviance is the generalized model’s residual sum of squares: −2 × log-likelihood, or twice the gap between the fitted model and a saturated one that fits every case exactly. Lower is better, and every comparison between generalized models is a difference of deviances. The null deviance (242.3 on 192 df for recommend on overall) is the intercept-only model’s, the residual deviance (176.5 on 191 df) the fitted model’s, and their difference, 65.8, is the likelihood-ratio χ² for the model as a whole. For a count model the residual deviance against its degrees of freedom is also a goodness-of-fit test: 782 on 192 df for the café’s visit counts is a model that does not fit, and the reason is below.

Null deviance is the deviance of the intercept-only model – the outcome with no predictor, the baseline every fit is measured from.

Residual deviance is the deviance of the fitted model – what the predictors left unexplained – and its drop from the null deviance is the model’s likelihood-ratio χ².

Deviance residuals are the generalized model’s residuals on the deviance scale: each case’s signed contribution to the deviance, so that they behave like a linear model’s standardized residuals – roughly normal, mostly within ±2, and plotted against the linear predictor to look for pattern. The largest for recommend on overall is 2.26 in absolute value.

Pearson residuals are the alternative, $(\text{observed}-\text{fitted})/\sqrt{\text{variance}}$ with the variance the one the model assumes at that fitted value: the raw gap in units of the spread the model expects there. Their sum of squares is the Pearson χ² of the goodness-of-fit card and, divided by the residual degrees of freedom, the dispersion below.

Separation is a predictor, or a combination of them, that predicts the outcome perfectly for some of the cases: everyone on one side of a cut said yes, or a whole category of a predictor holds no noes at all. The likelihood then has no maximum – the coefficient that fits those cases best is infinite – and what the fitting routine returns when it stops is a huge estimate with a far huger standard error and a p-value near 1. The café’s 28 members who gave the top overall score of 10 all recommend it, where the other 165 split 62 to 103; put “gave the top mark” into a model of recommend beside taste and its coefficient comes back as 17.3 with a standard error of 1,201, p = .989 – a p-value that says nothing about the predictor and everything about the fit. Complete separation is a cut that predicts every case; quasi-complete, as here, is a subset predicted perfectly with the rest overlapping. A categorical predictor with a level that holds one outcome only is the usual cause, a small sample with a strong predictor the next, and the coefficients card warns when it detects either kind and names the terms.

Firth’s penalized likelihood is the remedy: a small penalty on the likelihood (Jeffreys’ prior) that pulls every coefficient a little toward zero, removes the small-sample bias of maximum likelihood, and – because the penalized likelihood has a maximum at a finite value however the data are split – keeps every estimate finite under separation. On the same model the “gave the top mark” coefficient becomes 2.83 (SE 1.48), p = .055, an odds ratio of 17: a member who gave the top mark has seventeen times the odds of recommending, an estimate with an interval rather than a shrug. On unseparated data it changes little – the slope of recommend on overall goes from 0.848 to 0.828 – which is why it is a defensible default for small samples and rare outcomes and not only a rescue. The app’s Firth estimation method is this fit, for binomial models.

Ordinal logistic regression models an outcome with a few ordered levels – a 1-to-5 rating, low/medium/high – without pretending the levels are equally spaced numbers. It places the levels on a hidden continuous scale cut into bands by thresholds, and models the odds of being above any cut as a linear function of the predictors: one slope per predictor, shared by every cut, which is the proportional-odds assumption. The café’s “The service was slow” item (1 = strongly disagree … 5 = strongly agree) on the wait and the branch, for the 185 members with a wait, gives 0.196 per minute (SE 0.051), p < .001, and 0.07 for Station, p = .79; exponentiated, an odds ratio of 1.22 [1.10, 1.34] – each minute waited multiplies the odds of a higher slowness rating, above whichever cut is chosen, by 1.22 – and 1.08 [0.63, 1.84] for Station. Undone through the cuts, a Riverside member who waited two minutes has a .15 chance of agreeing at all (a rating of 4 or 5) and .04 of strongly agreeing; at ten minutes, .46 and .16. The app’s Ordinal logistic type is this model – the proportional-odds or cumulative logit model – and asks for an ordinal outcome.

Thresholds are the cut-points between adjacent levels on the model’s hidden scale – four of them for a five-level rating – printed on the card as Thresholds (intercepts), each labelled with the pair of levels it separates. They stand where a logistic model’s single intercept stands, and like an intercept they anchor the scale rather than answer a question: the slowness model’s are 0.21 (1|2), 1.12 (2|3), 2.13 (3|4) and 3.59 (4|5) on the log-odds scale, so a member with no wait at all and Riverside’s branch has odds of e^−0.21 = 0.81 of being above “strongly disagree” and e^−2.13 = 0.12 of being above “neutral”. Their p-values test whether each cut sits at the scale’s zero, which is a convention rather than a hypothesis – the reason the card describes where a cut-point sits instead of calling it significant.

Proportional odds is the assumption that the predictors move the whole rating the same way at every cut: the odds ratio for being above “strongly disagree” is the odds ratio for being above “agree”. One slope per predictor is what makes the model readable, and it is a claim about the data, which can fail – most often when a predictor spreads a distribution rather than shifting it. The café’s Station members are both more often at “strongly disagree” (33 of 90, against Riverside’s 25 of 95) and more often at “strongly agree” (12 against 3), so the single Station slope of 0.07 – no difference – is the average of a negative effect at the low cuts and a positive one at the high cuts. Fitting the four cumulative splits one at a time gives Station slopes of −0.64 above 1, 0.06 above 2, 1.04 above 3 and 1.27 above 4, which is the picture the one number hides. The Brant test is the check, and the multinomial model the fallback.

Brant test compares the slopes the cumulative splits give separately with the one slope the model imposes – one χ² per predictor and an omnibus over all of them – and a small p says the predictor’s effect differs across the cuts. For the slowness model the wait passes, just: χ²(3) = 7.6, p = .054; the branch does not, χ²(3) = 18.9, p < .001, and the omnibus χ²(6) = 26.5, p < .001. A failure does not invalidate the model; it says which term’s single odds ratio is not to be trusted, and the answers are to read that predictor’s splits one by one, to refit the rating as a multinomial model that gives each level its own coefficients, or to keep the single slope as a summary and say so in the report. The app prints the test on the goodness of fit card of every ordinal model with two or more cuts.

Multinomial logistic regression models an outcome with several unordered levels – which of three products, which ward a patient went to after surgery, which candidate – as one logistic equation per level against a reference outcome: with J levels there are J − 1 equations, each with its own intercept and its own coefficient for every predictor, so two predictors and five levels cost twelve numbers where the ordinal model cost six. Nothing in the café is a choice among unordered options, but the model is also the fallback when an ordered rating fails proportional odds: the slowness rating on the wait and the branch, with “strongly disagree” as the reference, fits four equations, one per other level, and lets the branch’s effect differ level by level. The app’s Multinomial logistic type is this model; it accepts a numeric outcome with three or more distinct values for exactly that reason.

Reference outcome is the level every multinomial equation is measured against – the baseline the other levels’ odds are set beside, as a reference category is for a predictor. The default is the first level in sort order; the app’s reference outcome selector changes it, and changing it changes what every coefficient means while leaving the fit and the predicted probabilities as they were. Choose the comparison the question wants – the control condition, the status quo, the usual answer.

Relative risk ratio – RRR – is the exponentiated multinomial coefficient: the factor by which the probability of this level relative to the reference multiplies per unit of the predictor, the odds ratio of one equation. For the slowness rating against “strongly disagree”, Station’s ratios are 0.31 [0.13, 0.75] for “disagree”, 0.29 [0.12, 0.69] for “neutral”, 1.23 [0.51, 3.01] for “agree” and 2.05 [0.49, 8.58] for “strongly agree”: a Station member is a third as likely as a Riverside member to sit in the middle rather than at the bottom, and twice as likely to sit at the top – the spread the ordinal model’s single 1.08 averaged into nothing. The wait’s are 1.30, 1.24, 1.22 and 1.54 per minute, close to the one slope the ordinal model gave it. The price is the parameters – twelve against six, which AIC nevertheless prefers here, 547.5 against 563.0 – and it is paid in events per parameter. The card’s RRR and RRR 95% CI columns are these.

Events per parameter is what limits a logistic or count model, rather than the number of rows: the information about a coefficient comes from the cases in the rarer outcome – or, for a count, from the events counted – and the classic rule is at least ten of them per parameter estimated. recommend has 65 noes among the 200 members, so a model on the six satisfaction items and the branch, seven parameters, has 9.3 per parameter and the app says so as it runs; the slowness rating’s rarest level, “strongly agree”, holds 15 members, which is 7.5 per parameter for the ordinal model’s two slopes and 1.9 for the multinomial’s eight – the same data, four times the parameters, a quarter of the support for each. Below the line the estimates are still computed, and increasingly unstable: wide intervals, coefficients that swing with a case or two, and separation at the end of the road. The count families count their events as the sum of the counts – the 1,450 visits behind the Poisson model of visits_after make the check moot – and the app skips it for regularized fits, which are built for that regime.

Poisson regression models a count by the log of its expected value as a linear function of the predictors, with the Poisson distribution – whose variance equals its mean – for the scatter around it. The café’s visits in the eight weeks after the change, on branch and age, give a log-count 0.219 higher at Station and 0.011 higher per year of age.

IRR – the incidence rate ratio – is the exponentiated count coefficient, the factor by which the expected count multiplies per unit of the predictor, as the odds ratio is for a logistic model: e^0.219 = 1.24 [1.12, 1.38] for Station, so Station members visit 24% more often than Riverside members of the same age; 1.011 per year of age. The card’s IRR and IRR 95% CI columns are these.

Overdispersion is a count outcome more spread out than the Poisson distribution allows – a variance larger than the mean – and it is the rule rather than the exception in real counts, where cases differ in ways the predictors do not capture. The after-change visits have a mean of 7.6 and a variance of 39.2; Station’s regulars alone put the branch’s variance at 57.7 against a mean of 8.5, where Riverside’s is 20.0 against 6.8. A Poisson model fitted to such data gets the coefficients about right and their standard errors far too small, so that everything looks significant; the fix is a model with a spread of its own.

Dispersion is the diagnostic: the Pearson χ² over its degrees of freedom, which a Poisson model that fits has near 1. The café’s is 4.69 – the counts are nearly five times as variable as the model assumes – and the app’s score test beside it (z = 3.50, p < .001) confirms it, with the advice to try the model below.

Negative binomial regression is the Poisson model with an extra parameter, θ, for the spread the predictors leave unexplained; it fits the same log-linear equation and reports the same rate ratios. On the café’s visits θ = 2.49 (SE 0.34), the AIC drops from 1474 to 1156, and the coefficients barely move – Station 0.222 against 0.219 – while their standard errors double, from 0.053 to 0.106 for the branch and 0.003 to 0.005 for age, with p going from < .001 to .036 and .039. The evidence was always this thin; the Poisson model had overstated it. The app’s Negative binomial type is this model, and the two-part count models below handle the further case where the surplus spread is a pile of excess zeros.

Excess zeros are more zeros in a count than the count model expects – the members who never came back, the patients with no complaints, the days with no sales. Whether the zeros are excess is a comparison, not a count. Eight of the 195 members with an age made no visit in the eight weeks after the change; the Poisson model of visits_after on the branch and the age expects 0.2 of them – it cannot produce the zeros it sees – while the negative binomial expects 6.7, because its wider spread reaches zero on its own. A pile of zeros is one face of overdispersion, so the negative binomial is tried first, and the two-part models below are for the case where its expected zeros still fall well short of the observed ones. The app’s goodness of fit card for a two-part model prints the observed zeros against those expected by the model and by its one-equation counterpart, with the ΔAIC and ΔBIC that say whether the second equation earned its keep.

Two-part count model is a count model with a second, logistic equation for the zeros: one set of predictors for the counts and one – any subset of them, or a constant alone – for the zero side, fitted together. The app’s two-part count models are the two below, each with a Poisson or a negative binomial on its count side; they differ in what their zero equation is about.

Zero-inflated model says the sample is two kinds of case: an always-zero group that never generates a count, and the rest, whose counts follow a Poisson or negative binomial that can produce zeros of its own – so a zero can come from either source, and the zero equation models the probability of being an always-zero case. On the café’s visits, with a negative binomial on the counts and a constant zero probability, the always-zero share comes out at 1.3% – two or three members – the count side’s rate ratios barely move from the negative binomial’s (Station 1.26 against 1.25, age 1.010 against 1.011), and the expected zeros rise from 6.7 to 8.3 at the price of one parameter: AIC 1157.4 against 1156.0, BIC 1173.7 against 1169.1. The card’s Excess zeros row reads the gap between 6.7 and 8 as a shortfall; its Model choice row reads the equation that closes it as not worth its parameter, and with a gap of a zero and a third the second is the verdict to carry. Choose it when a genuinely immune group is believable – non-smokers asked how many cigarettes they smoke a day, members who moved away.

Hurdle model splits the count the other way: first whether a case clears zero at all – a logistic equation for any against none – and then, for the cases that did, how many, from a count distribution with the zero cut out of it. Every zero belongs to the first equation, so the model reproduces the observed zero count exactly (8 of 8 here), and its zero equation is a model of participation: 95.9% of members cleared the hurdle, and putting the branch into that equation gives Station a log-odds of −0.56 (SE 0.74), p = .45 – no sign the branches differ in who comes back at all. The count side is for the members who came: rate ratios of 1.30 for Station and 1.010 per year of age on a negative binomial with θ = 2.67, AIC 1157.1 – again one parameter for no gain over the plain negative binomial. Choose it when the first unit is a different decision from the rest – seeing a doctor at all against how often. The two families’ zero equations point opposite ways: zero-inflation models being a zero and the hurdle models clearing one, so the same sign means the opposite thing, and each card says which its equation is.

The café’s zeros, then, are ordinary, and the two-part fits say so by costing a parameter and buying nothing. Their opposite number is the zero-inflated Poisson, a common reach for a count with visible zeros: on these visits it expects 8.0 zeros and scores AIC 1422 – better than the Poisson’s 1474, far worse than the negative binomial’s 1156 – because it has fixed the zeros and left the overdispersion where it was. The order of business for a count is the dispersion first, the zeros second, and a two-part family when the second problem survives the fix for the first.

Classifying with a model #

A logistic model returns a probability for every case; a decision – recommend or not, treat or not, flag or not – needs a yes or a no, and turning the one into the other is a choice the model does not make on its own. This section is how that choice is made and how a classifier is judged, on the café’s model of recommend on overall: 193 members with an overall score, 131 of whom recommend the café and 62 who do not, and a fitted probability of recommending that runs from .01 at a score of 1 to .97 at a score of 10. Because the score takes whole numbers only, the model produces ten distinct probabilities, and every rule below comes down to “call a member a recommender from a score of k up”.

Classification threshold is the probability above which the model’s answer is yes: a case whose fitted probability clears it is predicted positive, the rest negative. One half is the natural-looking choice and rarely the right one: it treats a missed positive and a false alarm as equally bad, and where the sensible cut lies moves with how common the positives are. For the café, one half means “a score of 7 or above” – the probabilities either side of it are .48 at 6 and .69 at 7 – and it catches 84% of the recommenders while ruling out 63% of the others; raise the bar to a score of 8 (any threshold between .69 and .84, which the app reports as their midpoint, .76) and the two figures become 66% and 90%. Every threshold trades the one for the other, which is why the app’s ROC analysis looks at all of them and picks one by a stated rule – Youden’s J by default – rather than settling on one half.

Classification accuracy is the share of cases the model puts in the right class at a given threshold: .772 for the café’s model at one half, 149 of 193, against a no-information rate of .679. It hides which way the errors go – a model can be 90% accurate on a rare outcome by never predicting it – and it is not what the threshold rules maximise: at the app’s own threshold of .76 the café model’s accuracy is .741, lower than at one half, because the rule that chose the cut was balancing the two kinds of error rather than counting correct calls.

No-information rate is the accuracy of the best guess that uses no predictor at all – always the most common class: .679 here, the share of members who recommend the café, so a model has to beat it before “77% correct” means anything. The app’s Accuracy > no-information rate (p) row is a one-sided binomial test of whether it does.

Confusion matrix is the 2 × 2 table of predicted class against observed at one threshold: at one half, 110 members correctly predicted to recommend (true positives), 39 correctly predicted not to (true negatives), 23 predicted to recommend who did not (false positives) and 21 predicted not to who did (false negatives). Every metric below is a ratio of its cells. For an ordinal or multinomial model the app prints it with one row and one column per level, the row being the level the model assigned, so that a level the model never predicts shows as an empty row rather than vanishing.

Sensitivity – the true positive rate, or recall – is the share of the actual positives the model catches: true positives over all observed yeses, 110 ⁄ 131 = .84 at one half and 87 ⁄ 131 = .66 at the app’s threshold of .76. It is the number a screening test is judged by, since a missed case is the error a screen exists to avoid, and it does not depend on how common the positives are.

Specificity – the true negative rate – is the share of the actual negatives the model correctly rules out: true negatives over all observed noes, 39 ⁄ 62 = .63 at one half and 56 ⁄ 62 = .90 at .76. Sensitivity and specificity pull against each other as the threshold moves – the café’s cut at a score of 8 buys 27 points of specificity with 18 of sensitivity – and, like sensitivity, it is a property of the model and the threshold alone, not of the prevalence.

Positive predictive value (PPV) – precision – is the share of the model’s positive predictions that are right: true positives over everything predicted positive, 110 ⁄ 133 = .83 at one half and 87 ⁄ 93 = .94 at .76. It answers the question the user of a prediction asks – the model says yes; how likely is that to be true? – and unlike sensitivity it depends on the prevalence: the same sensitivity and specificity give a lower PPV wherever positives are rarer, because the false alarms then come from a bigger pool.

Negative predictive value (NPV) is the share of the model’s negative predictions that are right: true negatives over everything predicted negative, 39 ⁄ 60 = .65 at one half and 56 ⁄ 100 = .56 at .76 – the price of the higher bar, since the 44 recommenders it now misses all land among the predicted noes. It moves with prevalence the opposite way from PPV: rarer positives make a negative prediction safer.

Prevalence is the share of positives among the cases – .68 of the café’s members recommend it – and it is what PPV, NPV and accuracy are computed at: the app derives the three from the sensitivity, the specificity and the prevalence rather than from the sample’s counts, so that a different prevalence can be entered. If the café’s customers at large recommended it at 30% rather than the loyalty members’ 68%, the same model at the same threshold would have a PPV of .75 instead of .94 and an NPV of .86 instead of .56, with the sensitivity, the specificity and the AUC unchanged; the population class prevalences option is for exactly that case – a sample whose base rate is not the population’s, a case-control study above all.

ROC curve is the picture of every threshold at once: sensitivity up the side against 1 − specificity – the false positive rate – along the bottom, one point per threshold, joined. The café’s has ten points, one per score, from the corner where everyone is called a recommender (sensitivity 1, specificity 0) to the corner where no one is, and the point for “8 or above” sits nearest the top-left corner, where a perfect classifier would be. The diagonal is a coin toss – a model whose probabilities carry no information about the outcome – and the further the curve bows toward the top-left, the better the model separates the two classes, at whatever threshold is finally chosen.

AUC – the area under the ROC curve – is a classifier’s quality freed from any one threshold: the probability that a randomly chosen positive is given a higher predicted probability than a randomly chosen negative. .5 is a coin toss and 1 is perfect separation; the café’s model scores .834, with a 95% interval of [.78, .89] – pick a recommender and a non-recommender at random and five times in six the model rates the recommender higher. It measures discrimination, the ranking, and nothing about whether the probabilities themselves are right, which is calibration; it is the same number as the common language effect size of the predicted probabilities between the two groups; and the conventional bands read .7 to .8 as acceptable, .8 to .9 as good and above .9 as excellent. The app reports it with a DeLong or a bootstrap interval and, on request, cross-validated.

DeLong’s test compares two AUCs computed on the same cases. Two classifiers scored on one sample are not independent – a member who is easy to rank for one is usually easy for the other – so the difference between their AUCs wanders less from sample to sample than two unrelated AUCs would, and a test that ignored the overlap would understate the evidence. DeLong’s method writes each AUC as an average over the cases of how often each positive outranks the negatives, which gives it a standard error without resampling and, for two AUCs on the same cases, their covariance; the difference divided by its standard error is a z. On the café data, overall and taste as separators of who recommends the café, on the same 193 members: AUC .834 [.78, .89] against .712 [.64, .79], a difference of .122, z = 3.08, p = .002 – the overall score ranks the recommenders better than the taste item alone does. The same standard error is the DeLong option of the AUC confidence interval selector in regression and comparison analysis; the test itself is comparison’s Compare AUCs (pairwise DeLong’s test) and, in regression’s model comparison, the p (vs best) of each candidate’s curve against the top model’s. It needs every curve on the same cases, so a case missing on any of the variables compared is dropped from all of them.

Youden’s J is sensitivity + specificity − 1 – the height of a ROC point above the diagonal – and the app’s default rule for choosing the threshold: the cut that maximises it treats a missed positive and a false alarm as equally costly. For the café it is .57 at a score of 8 or above (sensitivity .66, specificity .90), against .47 at 7 or above, the cut one half implies. The Optimal threshold rule selector offers two alternatives, closest to (0, 1) and cost-weighted. Whichever rule chose it, the threshold was chosen on the same cases it is scored on, so its metrics are optimistic, and the app’s intervals re-choose the threshold inside every resample to price that.

Closest to (0, 1) is the threshold rule that picks the ROC point nearest the top-left corner – the perfect classifier’s, with sensitivity 1 and a false positive rate of 0 – by straight-line distance, $\sqrt{{(1-\text{sensitivity})}^2+{(1-\text{specificity})}^2}$. Where Youden’s J adds the two shortfalls, this squares them, so it prefers two moderate shortfalls to one small and one large and can settle on a more balanced point when the curve is lopsided. On the café’s model the two rules agree – a score of 8 or above, threshold .76, sensitivity .66 and specificity .90 – as they usually do on a curve with one clear shoulder. It is an entry of Optimal threshold rule in both modules that draw a ROC curve.

Cost-weighted is the threshold rule for when the two errors do not cost the same. It maximises the sum Youden’s J maximises with the specificity term weighted by $(1-\text{prevalence})/(\text{cost}\times \text{prevalence})$, where cost is how many false alarms one missed positive is worth – the Cost of a missed positive relative to a false alarm in regression, the Cost asymmetry ratio in comparison – and the prevalence is the sample’s or the one entered, so a costly and common positive pulls the threshold down and a costly false alarm pushes it up. Told that a missed positive costs three false alarms, among the café’s 68% of recommenders, the rule weighs specificity at about a sixth of sensitivity and drops the cut to a score of 4 or above (threshold .11): every recommender is caught and one non-recommender in ten ruled out – the cut for a screen that must miss no one, where Youden’s J gave the cut for a diagnosis. Because a ratio can be meant either way round, the app reports both thresholds – the one for a missed positive as the costlier error and the one for a false alarm – and the second lands back at .76 here; at a ratio of 1 the rule is Youden’s.

F1 is the harmonic mean of PPV and sensitivity,

and reads .78 at the café’s threshold of .76, from a PPV of .94 and a sensitivity of .66. It scores the positive class alone – the true negatives appear nowhere in it – which makes it the metric for a rare outcome, where accuracy is dominated by the negatives the model gets right for free, and a poor guide when the positives are the majority: on the café’s data F1 is highest, .84, at a score of 6 or above, and calling every member a recommender still scores .81, because a generous threshold loses little precision when 68% of the calls are right by default. Maximising it is a threshold rule in its own right elsewhere; the app reports it at the threshold its rule chose and leaves the choosing to the rules above.

Brier score is the mean squared error of the predicted probabilities against the outcomes coded 0 and 1: for every case, ${(\text{probability}-\text{outcome})}^2$, averaged. Lower is better; 0 is a model that says 1 for every yes and 0 for every no, .25 is a model that says one half to everyone, and the café’s model scores .151. Unlike the AUC it charges for the probabilities being wrong as well as for their order – a model whose probabilities all huddle near one half can rank the cases perfectly and still score badly – so it is the summary of a classifier’s probabilities where the AUC is the summary of its ranking.

Brier skill is the Brier score rescaled against the forecast that costs nothing – always predicting the base rate, .68 for every member, which would score .218: 1 − .151 ⁄ .218 = .31, the share of that baseline’s error the model removes. 0 is a model no better than the base rate, 1 is perfect, and a negative value is a model whose probabilities are worse than saying “68%” to everyone. It is the number to compare across outcomes with different base rates, since a raw Brier score falls on its own as an outcome gets rarer.

Calibration is whether the probabilities mean what they say: among the cases the model gives a .7, do seven in ten turn out positive? The café’s model gives .69 to a member who scored 7, and 23 of the 40 members at that score recommend it – .58; it gives .84 to a score of 8, and 37 of the 39 there do – .95. The probabilities are a little too flat through the middle: the true curve is steeper than the fitted one. A model can be well calibrated and discriminate poorly – a base-rate forecast is perfectly calibrated and ranks no one – or discriminate well and be miscalibrated – double every log-odds and the ranking is untouched while every probability moves – so the two are checked apart: discrimination by the AUC, calibration by the Brier score and the tests below.

Hosmer–Lemeshow test checks calibration by sorting the cases by predicted probability, cutting them into g groups of about equal size – ten by default – and comparing, in each, the positives observed with the number the probabilities add up to; the gaps are summed as a χ² on g − 2 degrees of freedom, and a small p says the probabilities are off somewhere. For the café’s model the ten distinct probabilities allow only six groups, and the test gives χ²(4) = 9.89, p = .042 – driven by the scores of 7 and 8, where the observed rates are .58 and .95 against the fitted .69 and .84. Its known weakness is that the verdict depends on g: regroup the same cases and p can cross .05 either way, which is why the app lets the count be set, reports the count it actually used, and prints beside it the le Cessie–van Houwelingen test, which weighs the same squared gaps case by case with no groups at all – z = 0.87, p = .38 here. When the two disagree, as they do for the café, the departure is real but local, and what to read is where it sits – the two scores in the middle of the range – rather than either p.

Cross-validated AUC is the AUC computed on cases the model was not fitted to, and the honest estimate of how it will classify new ones. The sample is dealt into k folds, each holding the same share of positives; the model is refitted on all but one fold and that fold’s cases are scored, until every fold has been held out once; the AUC of each fold’s held-out predictions is averaged, and the deal is repeated – ten folds and ten repetitions by default – with the repetitions averaged in turn. The in-sample AUC was computed on the cases the fit had already seen, so it is optimistic by an amount that grows with the number of predictors: the café’s one-predictor model scores .834 in-sample and .831 cross-validated, because a single slope has almost nothing to overfit with, while a model of recommend on the six items, the wait, the age and the branch scores the same .834 in-sample on its 180 complete members and .790 [.72, .86] out of sample – the same headline number, a different model. The gap is the overfitting the in-sample number hides, and the interval on the cross-validated figure – from the influence curve of the estimator, widened by the spread between repetitions – is the one to report.

Reporting #

A regression is reported as its equation and its fit: the model (outcome, predictors, type, n and how the incomplete cases were treated), the overall fit (R² and adjusted R² with F and its df, or the pseudo-R² and likelihood-ratio χ² for a generalized model) and, per term, the coefficient with its standard error or interval and its test – “spend after the menu change rose by 0.72 per unit of prior spend, 95% CI [0.61, 0.82], t(197) = 13.4, p < .001; R² = .48” – with odds ratios or rate ratios for a generalized model in place of raw coefficients. Name the reference category of every categorical predictor (and the reference outcome of a multinomial model), the meaning of the intercept if it is read at all, and the diagnostics that were run with their outcome: what the residual plot showed, the Breusch–Pagan and RESET verdicts, whether robust standard errors were used and why, the largest VIF, and any case that was influential and what was done about it. A model chosen among several is reported with the alternatives and the criterion (AICc, cross-validation) that chose it – or, when several were about as good, with the model-averaged coefficients and the size of the confidence set. A model used as a classifier is reported with its AUC and interval, cross-validated where the sample allows, the threshold and the rule that chose it, sensitivity, specificity, PPV and NPV at the prevalence they were computed at, and the Brier skill with the calibration verdict. A logistic or count model adds its events per parameter and whether Firth’s correction was used and why; an ordinal model the Brant verdict; a two-part count model which family it is, which count distribution, and what entered its zero equation.

Where this appears in DataSuite #

• Regression analysis – every model on this page: the coefficients and model fit tables, the ANOVA table, correlations, collinearity, residual and influence diagnostics, goodness of fit; the logistic, ordinal, multinomial and count regression types with the two-part count models, Firth estimation and the ROC analysis; moderators and mediators; model comparison by AICc and Akaike weights, with its model-averaged coefficients and variable importance; robust, quantile and regularized fits.
• Correlation analysis – the scatterplots with their fitted line, the OLS diagnostics that run the same Breusch–Pagan and RESET tests on each pair, and partial and semi-partial correlations.
• Comparison analysis – ANCOVA, a regression with a factor and a covariate under a comparison’s vocabulary, and the homogeneity of regression slopes check, which is an interaction test; its classification (ROC) analysis draws a curve per dependent variable with the same threshold rules and compares the AUCs with DeLong’s test.
• Structural equation modeling – mediation drawn as a path model: serial chains, latent mediators, and the total, direct and indirect effects of every pair with bootstrap intervals.
• Time-to-event analysis – Cox regression, whose coefficients exponentiate to hazard ratios the way a logistic model’s do to odds ratios.
• Meta-analysis – meta-regression, the effect sizes of studies regressed on their characteristics.
• Time series analysis – the autocorrelation the Durbin–Watson test looks for, modelled rather than tested.
• Structural equation modeling – regressions among latent variables, several at once, with the same coefficients and standard errors.
• Data transformation – centering and standardizing predictors before a fit.
• Settings – the VIF thresholds, the correlation strength bands the zero-order column is read on, and the confidence level of every interval.