Survival analysis concepts #

Some outcomes are not a value but a wait: how long until a patient relapses, a machine fails, a customer cancels, a student drops out. What makes such data its own branch of statistics is that when the study ends, some of the waits are still going on – the patient has not relapsed, the machine still runs – and a method that ignores those unfinished waits, or treats them as if they had ended, gets the answer wrong. This page explains the ideas in order: censoring and its variants, the survival curve and how it is estimated, the hazard behind it, how two groups’ curves are compared, the models that put covariates on the hazard or on the time scale, and what changes when more than one kind of event can end the wait. The café data holds no time-to-event column, so the numbers on this page come from a study every survival textbook uses: 26 women treated for ovarian cancer in a trial reported by Edmunson and colleagues in 1979, shipped with R’s survival package as ovarian. Each row is one patient: her follow-up in days, whether she had died by the end of it, her age, whether residual disease remained after surgery, and which of two chemotherapy regimens (A or B) she received. Copy the table below into a spreadsheet, save it as CSV and load it into the time-to-event analysis module: pick days as the time variable and status as the event variable – the module pre-assigns alive to Censored and died to Event – and treatment as the grouping variable; every number on this page is reproduced with the other options at their defaults.

Twelve of the 26 died during follow-up; the other fourteen were alive when last seen, between 377 and 1227 days after entering the trial.

Time and censoring #

Time-to-event data records, for each subject, how long they were followed and whether the event of interest happened at the end of that time. The outcome is therefore two columns, not one: a time variable – a non-negative duration from a common starting point, in whatever unit the study used – and an event indicator that says how each subject’s follow-up ended. A survival analysis needs both, and in the app they are the two roles every method asks for first. In the ovarian trial the time variable is days since entering the trial and the event indicator is status, died or alive. A nurse: days from admission to discharge → time variable, discharged or still on the ward → event indicator, ward → grouping variable; an engineer: hours to failure → time, failed or still running when the test stopped → event, supplier → grouping; a marketer: days until a subscriber cancels → time, cancelled or still subscribed → event, plan → grouping, age and monthly spend → covariates.

Survival analysis is the family of methods for time-to-event data. The name is a relic of its origin in mortality tables; the event can be anything that happens once, and “survival” means only “not having had it yet”. What sets the family apart is censoring: a study’s follow-up ends before every subject has had the event, and the unfinished waits carry real information – the subject lasted at least that long – that an ordinary mean or regression cannot use. Averaging the observed times treats a patient alive at 1227 days as if she had died then; dropping the unfinished waits throws out exactly the subjects who did best. Survival methods count every subject for as long as they were observed and no longer, and report the answer as a survival curve, a hazard, or a model of how covariates move one of the two.

Censoring is the incomplete observation of a subject’s time. In its usual form – right censoring – the subject was followed to some last time and had not had the event by then, so the true time is known only to be greater than the time recorded. Fourteen of the 26 women are censored: the woman last seen alive at 1227 days may have lived years longer, and the table can only say her time exceeded 1227. Censoring comes from a study that ends before everyone has had the event, from subjects who move away or withdraw, and from subjects who leave for a reason unrelated to the event. A censored subject counts among those at risk up to her last time and then leaves the count without being counted as an event – which is how every method below uses her. The methods assume the censoring is non-informative: that a subject censored at time t was, at that moment, no more and no less likely to have the event than the subjects who stayed. When that fails the analysis is biased, and there is no repair in the data – see informative censoring.

Interval censoring is knowing only that the event happened between two times – the last assessment at which it had not happened and the first at which it had – rather than exactly when. It is the natural state of anything detected at scheduled visits: a tumour found on the annual scan appeared some time in the past year; a part found broken at the weekly inspection failed some time that week. Treating the event as though it occurred at the visit that found it pushes every time late and biases the survival curve up; treating it as the midpoint guesses. Interval-censored methods keep the bracket and estimate the curve from the brackets directly. In the app the time variable holds the lower end of each bracket – the last event-free assessment – and a second upper bound variable holds the upper end, left empty for subjects who never had the event and are therefore right-censored at their last assessment.

Left truncation – also delayed entry – is when a subject enters observation some time after the clock that defines the time variable started, so that they could not have been observed to have the event before they entered. Anyone who had the event before that point never made it into the data at all. Retirees joining a registry at different ages, patients referred to a clinic some months after diagnosis, and any analysis that uses age as its time scale have this shape. Ignoring it counts subjects as at risk during a window in which they could not have been observed failing, and inflates the survival curve. The fix is to count each subject in the risk set only from their entry time on; in the app that is the entry-time variable, which must be strictly earlier than the subject’s event or censoring time – a subject who enters and leaves at the same moment was at risk for no time and tells the analysis nothing.

Informative censoring is censoring that is related to the event – patients who drop out of a trial because they are getting worse, machines withdrawn from a test because they are starting to rattle. The subjects who leave are then not exchangeable with those who stay, and the survival curve estimated from the stayers is too optimistic, because it silently assumes the leavers would have fared as well as they did. No survival method in the app, or anywhere, can detect this from the data alone: the assumption is about the reason for censoring, which the data does not record. It is checked by knowing the study – why did people leave, and could that be related to the outcome? – and reported in the write-up, with a sensitivity analysis when the answer is uncomfortable.

The survival curve #

Survival function – written S(t) – is the probability of still being event-free at time t: the share of subjects who would still be waiting if every subject were followed that long. It starts at 1 at time zero, never rises, and steps down each time an event occurs. Reading it at one time gives a survival probability: for the women on regimen A the estimated S(365) is 0.615 – about 62% were alive one year in – and for regimen B it is 0.846. Reading across gives the time at which the curve crosses a probability, such as the median. The curve is the standard summary of time-to-event data because it uses censored subjects correctly: each contributes to the estimate for exactly as long as she was watched.

Kaplan–Meier estimate – the product-limit estimate – is the standard way to compute a survival curve from censored data. It walks through the event times in order and, at each, multiplies the survival so far by the share of those still at risk who did not have the event:

where $n_i$ is the number of subjects at risk just before time $t_i$ – not yet failed, not yet censored – and $d_i$ the number who failed at it. The thirteen women on regimen A start at S = 1. The first death, at 59 days with 13 at risk, takes it to 12⁄13 = 0.923; the deaths at 115, 156, 268, 329 and 431 days, each with one fewer at risk, take it to 0.846, 0.769, 0.692, 0.615 and 0.538. The women censored at 448 and 477 days then leave the risk set without a step. At 638 days five women are at risk and one dies, so S becomes 0.538 × 4⁄5 = 0.431, and the four censored after that produce no further steps – the curve ends flat at 0.431. A censored subject is thus used exactly for what she tells: she raised $n_i$ at every event before her last time, and nothing after. The pointwise confidence interval around the curve widens as $n_i$ falls; the app draws it as a band and lets you switch it off.

Number at risk is $n_i$ – how many subjects are still being watched and still event-free just before a given time. It is what every estimate at that time rests on, and it shrinks throughout follow-up as subjects fail or are censored, which is why the right-hand end of a survival curve is the least reliable part of it. Regimen A had 13 women at risk at day 0, 8 at one year, 4 at two years and one at 1106 days; a step there would have taken the curve to zero on the strength of one woman. The app prints the number at risk under the survival plot at each axis tick, one row per group, so the reader can see how much of the curve to believe – and greys out a requested survival probability that falls beyond a group’s last observation, since the value there is the last estimate carried forward, not something observed.

Median survival time is the time at which the survival curve first reaches 0.5: the point by which half the subjects are expected to have had the event. It is the usual single-number summary of a survival curve because, unlike the mean, it does not need the curve to reach zero – it needs only half the subjects to have failed. For regimen A the curve crosses 0.5 at 638 days, so the median is 638; for regimen B the curve never drops below 0.564, so the median is not reached, and the app prints – rather than a number. A median can carry a confidence interval: for regimen A the lower limit is 268 days and the upper is undefined, because the upper confidence band never falls to 0.5. A study in which fewer than half the subjects have had the event has no median to report, which is one reason to reach for the restricted mean.

Restricted mean survival time (RMST) is the area under the survival curve from time zero to a chosen horizon τ – the average event-free time over that window, in the time variable’s units. It is defined whether or not the curve reaches 0.5, it uses the whole curve rather than one crossing point, and its meaning is plain: “over the first three years, patients on B were event-free for 171 days longer on average”. The horizon has to be chosen, and the app takes the largest time that every group actually reaches, so no group’s curve is read past its own follow-up: for the ovarian trial τ = 1106 days, the last observation on regimen A. Over that window regimen A averages 649.6 event-free days and regimen B 820.3; the difference, −170.7 days for A relative to B, carries a confidence interval from −474.0 to 132.5 and a p of .27 – a difference of half a year that the 26 women cannot pin down. Two RMSTs are compared only at the same τ, which is why the app reports one horizon for the whole card.

Hazard – h(t) – is the instantaneous rate of the event at time t among the subjects who have not yet had it: how likely a subject who has lasted until t is to fail in the next moment. It is a rate, not a probability, so it can exceed 1 when the time unit is coarse, and its shape over time is what distinguishes one process from another – a hazard that rises with time (ageing, wear), falls (early failure of the weak, then stability), or stays flat (a memoryless process, where having lasted so far predicts nothing). The survival function and the hazard carry the same information in two forms: S(t) is what has accumulated, h(t) is the rate at which it is being spent. Every survival model in the app describes one or the other – a Cox model puts covariates on the hazard, an accelerated failure time model on the time scale, and the parametric families each commit to one hazard shape. The Weibull fit to the ovarian trial estimates a shape of 1.81, an increasing hazard: the risk of death rose the longer a woman had been in the trial.

Cumulative hazard function – H(t) – is the hazard added up from time zero to t, and it is tied to the survival function by $S(t)=e^{-H(t)}$, so a cumulative hazard of 0.69 means half the subjects have had the event. Its estimate from censored data is the Nelson–Aalen sum of, at each event time, the events divided by the number at risk:

For regimen A up to 638 days that is 1⁄13 + 1⁄12 + 1⁄11 + 1⁄10 + 1⁄9 + 1⁄8 + 1⁄5 = 0.787. Exponentiating it gives a second survival estimate, $e^{-0.787}$ = 0.455 against the Kaplan–Meier 0.431 – the Fleming–Harrington estimate, which is always a little higher than Kaplan–Meier and closer to it the larger the risk sets are; the two agree to the third decimal on a few hundred subjects. The app plots the Nelson–Aalen curve beside the survival curve: a straight line means a constant hazard, a curve bending upward an increasing one, and a bend downward a decreasing one, which is the quickest visual check on which parametric family is plausible.

Comparing groups #

Log-rank test compares the survival curves of two or more groups under the null hypothesis that they are identical. At each event time it asks how many events each group would have contributed if the groups shared one hazard, given how many of each were at risk, and sums the differences between those expected counts and the observed ones over the whole follow-up; the sum, squared and scaled, follows a χ² distribution on one degree of freedom fewer than there are groups. For regimens A and B it gives χ²(1) = 1.06, p = .30: seven deaths among thirteen against five among thirteen is well within what one hazard shared by both would produce. The test says whether the curves differ, not by how much – that is the hazard ratio’s job – and it is most powerful when the hazards are proportional, one group’s a constant multiple of the other’s throughout follow-up; curves that cross are the case it is weakest on. Every death and every censoring time counts, so the test uses the whole follow-up rather than the share alive at one date, which is why it is preferred to comparing survival at a single time point.

Weighted log-rank tests are the log-rank test with the event times weighted unequally, so that differences early or late in follow-up count for more. The family the app offers is indexed by a number ρ: ρ = 0 is the plain log-rank test, every event weighted alike; ρ = 1 weights each event time by the pooled survival estimate just before it – the Peto–Peto test, which puts most weight on early events, when survival is still near 1 and most subjects are at risk; ρ = 0.5 is Fleming–Harrington’s middle ground. On the ovarian trial the Peto–Peto test gives χ²(1) = 1.68, p = .19, more than the log-rank’s 1.06, because the two curves separate early – five of A’s seven deaths came in the first year against two of B’s five – and converge later. The weight is chosen from the question, before the data is seen: a treatment expected to help most in the first months is an early-weight question, a long-term effect a late one. Picking whichever weight gives the smallest p after seeing them all is p-hacking under another name.

Modelling the hazard #

Hazard ratio – HR – is how many times higher one subject’s hazard is than another’s, at every moment of follow-up: the exponential of a coefficient in a Cox or other proportional-hazards model. An HR of 1 is no effect; 2 means the event happens at twice the rate – not twice as often in total, and not twice as soon – and 0.5 at half the rate. For a binary covariate it compares the two groups; for a continuous one it is the ratio for a one-unit increase, so its size depends on the unit. In the ovarian trial a Cox model with age and treatment estimates an HR of 1.16 per year of age, with a 95% confidence interval from 1.06 to 1.27: a woman ten years older faced a hazard about ${1.16}^{10}$ ≈ 4.4 times as high. The HR for regimen B against A is 0.45 [0.13, 1.55]: the point estimate says B more than halved the hazard, the interval says the trial cannot rule out anything from an 87% reduction to a 55% increase. The confidence interval is read first, and whether it excludes 1 is the test.

Proportional hazards is the assumption behind the hazard ratio: that the ratio between two subjects’ hazards stays constant over the whole follow-up, so that one number can describe it. The Cox proportional hazards model builds on exactly this. It writes each subject’s hazard as an unspecified baseline hazard – the same for everyone and allowed any shape – multiplied by the exponential of a linear combination of the covariates, $h(t|x)=h_0(t)e^{{\beta }_1{x}_1+\cdots +{\beta }_k{x}_k}$. Because the baseline cancels out of the comparison between subjects, the model estimates the β without ever saying what the baseline looks like, which is why it is called semi-parametric: parametric in the covariate effects, free in the time shape. The price is the assumption itself. The app tests it with Schoenfeld residuals (cox.zph): for the ovarian model the age term gives χ²(1) = 0.47, p = .49, the treatment term χ²(1) = 0.47, p = .50, and the global test χ²(2) = 0.80, p = .67 – no sign that either effect changes with time. A significant test means the covariate’s HR is not one number but a curve, and the fitted HR is a weighted average of it over follow-up; the fixes are to stratify the baseline on the offending variable, to let its effect vary as a time-varying coefficient, or to switch to a parametric model whose time shape can be inspected. A stratified Cox model lets each level of a grouping variable have its own baseline hazard while sharing the coefficients, which removes that variable from the proportionality requirement at the cost of estimating no HR for it.

Time-varying coefficient is a covariate effect that is allowed to change over follow-up – the covariate’s value is fixed at baseline, but its hazard ratio differs from one period to the next. The usual way to fit one is piecewise: cut the follow-up into intervals, fit a separate coefficient for the covariate in each, and keep every other covariate proportional. It is the direct repair for a proportional-hazards violation on one term. A treatment that helps early and then wears off shows HRs drifting towards 1 across the periods – 0.28, 0.52, 1.03, say – where the proportional fit had reported one averaged 0.50 and hidden the pattern; a likelihood-ratio test of the piecewise fit against the proportional one is the formal test that the effect moves. The ovarian model shows no violation, so this repair is not called for there.

Time-dependent covariate is the other thing that sounds like the above and is not: a covariate whose value changes during follow-up – a treatment started mid-study, a lab value re-measured at each visit, an exposure that comes and goes. Fitting one requires the data in counting-process form, one row per subject per period the covariate held a value, and the app’s time-to-event module does not build that form; the R console does. The distinction matters because the tempting shortcut – defining a baseline group from something that happened later – manufactures immortal time bias.

Immortal time bias is the bias from classifying subjects by an event that happened during follow-up – “patients who eventually received drug X” against “patients who never did”. Everyone in the first group had, by definition, to survive long enough to receive the drug, so that stretch of their follow-up was time in which they could not die: they were immortal until classified. The comparison then credits the drug with survival that was a condition of being in the group. It is one of the most common errors in observational survival analysis and it can manufacture a large protective effect from nothing. The defence is to define groups from information available at time zero only, or to model the exposure as a time-dependent covariate that switches on when it actually happened.

Events per parameter limits a survival model the way it limits a logistic one: the information about a coefficient comes from the events, not from the rows, since a censored subject’s time bounds the answer without pinning it. The working rule is at least ten events per parameter estimated – a factor covariate spending one per level after the first, a parametric family one or two more for its shape parameters. The ovarian trial’s twelve deaths support one parameter comfortably and two on sufferance: the Cox model with age and treatment has six events per parameter, and its HR for treatment, with a confidence interval from 0.13 to 1.55, shows what that buys. Below one event per parameter the fit is singular – infinitely many coefficient sets fit equally well – and the app refuses it; between one and ten it warns. The regression basics page has the general rule.

Parametric models #

Accelerated failure time (AFT) models describe a covariate as stretching or compressing the time scale rather than multiplying the hazard: a subject with the covariate lives through the same survival curve as one without, but at a different speed. The model is a regression of the logarithm of the time on the covariates, $\log T={\beta }_0+{\beta }_1{x}_1+\cdots +\sigma \epsilon$, with the distribution of the error ε chosen from a family – Weibull, exponential, log-normal, log-logistic, Gaussian – that fixes the shape of the hazard. Committing to a shape is what makes the model parametric: it can extrapolate beyond the observed follow-up, predict a smooth curve, and be compared across families by AIC, and it is wrong when the shape is. The Weibull is the workhorse, because it is the one family that is both an AFT model and a proportional-hazards model: its hazard is monotone, rising when the shape parameter exceeds 1 and falling below it, and it reduces to the exponential – a constant hazard – at exactly 1. Fitted to the ovarian trial, the Weibull estimates a shape of 1.81 and a time ratio of 1.76 for regimen B; fitting all six families and ranking them by AIC puts the log-normal first at 183.4, the log-logistic 0.8 behind it, the Weibull 2.2 behind, the exponential 5.1, the Gompertz 5.8 and the Gaussian 15.0 – the last a poor description of a wait, since it allows negative times. Differences under two are no difference, so the top three are interchangeable here; a parametric curve is checked against the Kaplan–Meier curve before its effect is reported, whatever the AIC says.

Time ratio is the AFT model’s effect measure: the exponential of a coefficient, the factor by which the covariate multiplies the time to the event. A time ratio of 1.5 means the event comes 1.5 times later – a median of 400 days becomes 600 – and a ratio of 0.8 means 20% sooner. It is the mirror image of a hazard ratio, and often the more intuitive of the two, because it is expressed in the time variable’s own units: the Weibull fit to the ovarian trial gives regimen B a time ratio of 1.76 [0.91, 3.44] – deaths came, on average, three quarters again as late on B as on A, with an interval that includes 1 – and age a time ratio of 0.924 per year, so each additional year of age brought death about 8% sooner. The two measures are not interchangeable numbers: a Weibull hazard ratio equals the time ratio raised to minus the shape, so B’s time ratio of 1.76 at a shape of 1.81 is a hazard ratio of about 0.36, in the same direction as the Cox model’s 0.45.

Competing risks #

Competing risks are two or more kinds of event of which a subject can experience only the first: death from the disease and death from another cause, relapse and death without relapse, a customer cancelling and a customer being bought out. Once one has happened the others can no longer be observed, so they compete. A subject who had a competing event is neither censored in the ordinary sense – she did not leave the study still at risk, she left it for good – nor an event of the kind being studied, and treating her as either gives a wrong answer. In the app a competing-risks analysis is set up in the event indicator’s level roles: every non-censored level is given its own cause, and two or more causes unlock the methods below. The ovarian trial recorded one cause, death, so it has no competing risks; a study that had recorded death from cancer and death from other causes separately would.

Cumulative incidence function – CIF – is, for one cause, the probability of having had that event by time t in the presence of the others: the share of subjects who have failed from cause 1 by t, counting the ones who failed from cause 2 first as people who can no longer fail from cause 1. The temptation is to treat the competing events as censored and report 1 minus the Kaplan–Meier curve for cause 1, and it is wrong: that quantity is the incidence that would be seen if the competing events could be prevented and everyone who had one went on to be at risk of cause 1 – a hypothetical world – and it always overstates the real incidence. The CIFs of all the causes add up to 1 minus the probability of being event-free, which 1 − KM per cause never do. The app estimates CIFs by the Aalen–Johansen method and compares them across groups with Gray’s test, the competing-risks counterpart of the log-rank test, once per cause.

Cause-specific hazard is the hazard of one cause among the subjects still free of every cause – the rate at which, say, cancer deaths occur among those still alive. A Cox model fitted to it treats the competing events as censored, which is legitimate for this quantity: the question is what drives the rate of this cause among the survivors, and a subject who died of something else genuinely left the risk set. It answers the aetiological question – does this covariate change the biology of this cause? – and one such model is fitted per cause. What it does not answer is what happens to a subject’s overall chance of the event, because a covariate can lower the cause-specific hazard of cause 1 and still raise its cumulative incidence by lowering the hazard of cause 2 even more, leaving more people alive to fail from cause 1.

Subdistribution hazard is the hazard that governs the cumulative incidence function directly: the rate of cause 1 among the subjects who have not yet failed from cause 1 – including, in the risk set, those who have already failed from another cause and can never fail from this one. The Fine–Gray model puts covariates on it, and its subdistribution hazard ratio (sHR) says how a covariate shifts the eventual incidence of the cause, competing events and all. It answers the prognostic question – what does this covariate mean for a patient’s chance of this outcome? – which is the one a patient or a planner is asking. The two models can give different signs for the same covariate, and neither is wrong: they answer different questions, and a competing-risks report usually gives both.

Where this appears in DataSuite #

• Time-to-event analysis – the module: Kaplan–Meier and Fleming–Harrington curves with the log-rank family and RMST, Turnbull’s estimate for interval-censored data, Cox regression with its diagnostics and a time-varying coefficient, parametric AFT and Gompertz fits with an AIC ranking, and cumulative incidence, cause-specific and Fine–Gray models for competing risks.
• Regression basics – the coefficient, the reference level and dummy coding, the likelihood-ratio test and AIC, and the events-per-parameter rule, all of which a Cox or parametric fit inherits.
• Hypothesis testing – the p-value and χ² behind the log-rank, Gray’s and proportional-hazards tests, and the adjustment applied to pairwise comparisons.
• Confidence intervals – the pointwise band around a survival curve and the interval around a hazard ratio or a median.
• Outliers and missing data – the influence measures a Cox fit prints per subject, and the listwise deletion the module applies to its own roles.