Karanvir Singh
Independent researcher, Quantix Bio, Cambridge, MA, USA
Correspondence: karanvir.singh@quantixmind.com
Manuscript draft, August 2026. Not yet peer reviewed.
Background. Clinical risk models are usually validated once and deployed indefinitely, while the populations under them change. Which property of a model decays over calendar time, its discrimination or its calibration, determines the correct maintenance response, yet the two are rarely tracked separately over long horizons on stable national data.
Methods. A transparent logistic diabetes risk score (age, sex, body mass index, waist circumference, systolic blood pressure) was fit on the NHANES 1999 to 2002 cycles (8,201 adults aged 20 and over) and evaluated without modification on every subsequent two year cycle through 2017 to 2018 (3,933 to 5,296 adults per cycle; 45,412 participants in total). Diabetes was defined as glycohemoglobin of 6.5 percent or higher or self reported physician diagnosis. Each cycle we measured discrimination (AUC) and calibration (expected calibration error, ECE) on a held out test partition, and compared three maintenance strategies: never recalibrate, always recalibrate (isotonic regression on a 400 person holdout), and a prospective decision rule from our companion study that recalibrates only when holdout miscalibration exceeds 1.5 times the estimation noise expected at that holdout size. All analysis is in pure numpy, including a hand written reader for the SAS transport files NHANES distributes, and regenerates from public data with one command.
Results. Discrimination was essentially flat for fifteen years: AUC 0.785 in 2003 to 2004 and 0.769 in 2017 to 2018 (range 0.769 to 0.804). Calibration decayed almost monotonically and nearly fourfold: ECE rose from 0.016 in 2003 to 2004 to 0.057 in 2017 to 2018, tracking the rise in adult diabetes prevalence from 0.122 to 0.193 over the same period (correlation between cycle prevalence and miscalibration 0.96). The model increasingly underestimated risk while ranking patients as well as ever. Across the eight evaluation cycles, mean test ECE was 0.034 under never recalibrating, 0.028 under always recalibrating, and 0.026 under the decision rule, which declined to recalibrate in the three cycles where recalibration would have helped least or hurt.
Conclusions. Population drift attacked calibration while leaving discrimination intact, which is the failure mode a periodic AUC check cannot see. Model maintenance should monitor calibration explicitly, and a simple prospective rule, recalibrate only when measured miscalibration exceeds the estimation noise of the correction, outperformed both blanket strategies over two decades of real population change.
Keywords. calibration drift, temporal validation, transportability, NHANES, diabetes risk, model maintenance
A prediction model is validated at one point in time and then lives in a changing world. The populations it scores drift in demographics, treatment, and disease prevalence, and the literature on temporal transportability documents that transported models commonly fail in calibration before they fail in discrimination [1, 2, 3]. What is scarce is a long horizon measurement of the two properties separately, on a population sample designed to be comparable across decades.
NHANES provides exactly that instrument: a continuous national health examination survey with consistent measurement protocols, released in two year cycles since 1999, including laboratory glycohemoglobin [4]. Adult diabetes prevalence in the United States rose substantially over this period [5], making it a natural stress test: a model fit at 1999 era prevalence is asked to score the 2017 population.
This study fits a deliberately simple, fully transparent diabetes risk score on the 1999 to 2002 cycles and carries it, frozen, through 2017 to 2018, asking three questions. First, what decays, ranking or probability? Second, how does the decay relate to the underlying prevalence shift? Third, can the recalibration decision be automated prospectively, using the sample size aware rule from our companion study of recalibration harm [6], and does that rule beat the two default policies of never and always recalibrating?
NHANES cycles 1999 to 2000 through 2017 to 2018, adults aged 20 and over with complete age, sex, body measures, blood pressure and glycohemoglobin. Diabetes was defined as glycohemoglobin 6.5 percent or higher or self reported physician diagnosis, following the laboratory threshold in current diagnostic criteria [7]. Sample sizes per cycle ranged 3,933 to 5,296 (total 45,412; training cycles 8,201). Prevalence rose from 0.113 (training cycles) to 0.193 (2017 to 2018). NHANES files are distributed as SAS transport (XPT) files; a reader for the format, including the IBM hexadecimal floating point conversion, was written in numpy for this study and is released with it.
Logistic regression with L2 penalty on five standardized predictors: age, sex, body mass index, waist circumference, and systolic blood pressure (mean of first two readings). The model was fit once on the pooled 1999 to 2002 cycles and never refit. The deliberate simplicity keeps every coefficient inspectable and makes the drift interpretable.
Each later cycle was split into a 400 person holdout and a test partition. On the test partition we measured AUC, ECE (ten equal width bins) and Brier score for three strategies: never (frozen model), always (isotonic regression fit on the holdout, applied to the test partition), and rule (recalibrate only if holdout ECE exceeds 1.5 times the estimation noise an isotonic map is expected to introduce at that holdout size, interpolated from the calibrated model simulation of our companion study [6]). The rule sees only the holdout, never the test partition, so it is a genuinely prospective policy.
Reporting standard. This manuscript follows the TRIPOD+AI reporting guidance for clinical prediction model studies (title through limitations); the released code maps each reported number to the line that produces it.
The analysis is unweighted: NHANES sampling weights are not applied, so estimates describe the examined sample rather than the US population; the within survey comparison across cycles remains internally consistent, and a weighted replication is the natural extension. Fasting glucose is not used, so the outcome misses diabetes detectable only by fasting criteria. The 6.5 percent glycohemoglobin criterion was formally adopted in 2010, inside the study window; we apply it uniformly to all cycles as a fixed measurement rule rather than a clinical claim.
Table 1. Cohort characteristics, development pool versus final evaluation cycle.
| Characteristic | Development 1999-2002 | External 2017-2018 |
|---|---|---|
| N | 8,201 | 4,554 |
| Age, mean ± SD | 48.9 ± 18.5 | 51.2 ± 17.5 |
| Female | 52.6% | 51.1% |
| BMI, mean ± SD | 28.2 ± 6.1 | 29.7 ± 7.1 |
| Waist, cm, mean ± SD | 96.7 ± 15.0 | 100.8 ± 16.9 |
| Systolic BP, mean ± SD | 126.8 ± 21.5 | 126.8 ± 19.6 |
| Diabetes prevalence | 11.3% | 19.3% |
The population the frozen model faces in 2017 is older, heavier and nearly twice as diabetic as the one it was fit on; Table 1 is the drift mechanism in plain sight.
Table 1. The frozen 1999 to 2002 model evaluated on each later cycle.
| Cycle | n | Prevalence | AUC | ECE, frozen model |
|---|---|---|---|---|
| 2003-2004 | 3,933 | 0.122 | 0.785 | 0.016 |
| 2005-2006 | 4,119 | 0.119 | 0.786 | 0.016 |
| 2007-2008 | 4,953 | 0.156 | 0.784 | 0.032 |
| 2009-2010 | 5,296 | 0.145 | 0.804 | 0.031 |
| 2011-2012 | 4,592 | 0.156 | 0.783 | 0.046 |
| 2013-2014 | 4,955 | 0.149 | 0.792 | 0.030 |
| 2015-2016 | 4,809 | 0.172 | 0.778 | 0.049 |
| 2017-2018 | 4,554 | 0.193 | 0.769 | 0.057 |
AUC moved within a band of 0.035 over fifteen years with no meaningful trend. ECE rose almost monotonically from 0.016 to 0.057. The direction of the miscalibration is the expected one: as prevalence rose, the frozen model increasingly understated absolute risk while continuing to rank individuals essentially as well as on the day it was fit.
Table 2b. Bootstrap confidence intervals for discrimination, and logistic recalibration decomposition of the drift.
| Cycle | AUC (95% CI) | Calibration intercept | Calibration slope |
|---|---|---|---|
| 2003-2004 | 0.785 (0.766 to 0.802) | +0.094 | 1.040 |
| 2005-2006 | 0.786 (0.769 to 0.806) | +0.051 | 0.998 |
| 2007-2008 | 0.784 (0.768 to 0.799) | +0.382 | 1.055 |
| 2009-2010 | 0.804 (0.790 to 0.817) | +0.484 | 1.149 |
| 2011-2012 | 0.783 (0.766 to 0.799) | +0.428 | 1.015 |
| 2013-2014 | 0.792 (0.778 to 0.806) | +0.432 | 1.078 |
| 2015-2016 | 0.778 (0.763 to 0.793) | +0.415 | 1.009 |
| 2017-2018 | 0.769 (0.754 to 0.786) | +0.444 | 0.987 |
The decomposition is decisive about the drift's structure. The calibration slope stays within 0.99 to 1.15 across all eight cycles, while the calibration intercept climbs from +0.05 to about +0.44 and stays there: the frozen model's risk ordering and its logit scale remain correct, and the entire miscalibration is calibration in the large, exactly the failure an intercept only update repairs. This confirms quantitatively what section 4.3 argues mechanistically, and the bootstrap intervals show the flat AUC trajectory is not masking hidden discrimination loss.
The correlation between a cycle's diabetes prevalence and the frozen model's miscalibration on that cycle was 0.96. Calibration drift here is not a mysterious degradation of the model but the visible shadow of a population level epidemiological change that a fixed intercept cannot follow. This is precisely the calibration in the large failure that the transportability literature identifies as the first casualty of transport [1, 2].
Table 2. Standardized coefficients of the frozen model (log odds per standard deviation, fit on 1999 to 2002).
| Term | Coefficient |
|---|---|
| Intercept | -2.527 |
| Age | +0.729 |
| Female | +0.023 |
| Body mass index | -0.016 |
| Waist circumference | +0.693 |
| Systolic blood pressure | +0.123 |
Age and waist circumference dominate, systolic pressure contributes modestly, and body mass index adds almost nothing once waist is present, consistent with waist being the better adiposity signal for dysglycemia. The near zero sex and BMI terms are left in the model deliberately: the study freezes what a 2002 modeller would plausibly have shipped, not an optimized subset.
Table 3. Complete per cycle results: discrimination, Brier score, and the three maintenance policies.
| Cycle | n | Prevalence | AUC | Brier | ECE never | ECE always | ECE rule | Rule recalibrated |
|---|---|---|---|---|---|---|---|---|
| 2003-2004 | 3,933 | 0.122 | 0.785 | 0.0957 | 0.0157 | 0.0269 | 0.0157 | no |
| 2005-2006 | 4,119 | 0.119 | 0.786 | 0.0938 | 0.0158 | 0.0226 | 0.0158 | no |
| 2007-2008 | 4,953 | 0.156 | 0.784 | 0.1160 | 0.0317 | 0.0071 | 0.0071 | yes |
| 2009-2010 | 5,296 | 0.145 | 0.804 | 0.1071 | 0.0307 | 0.0314 | 0.0314 | yes |
| 2011-2012 | 4,592 | 0.156 | 0.783 | 0.1170 | 0.0456 | 0.0245 | 0.0245 | yes |
| 2013-2014 | 4,955 | 0.149 | 0.792 | 0.1103 | 0.0297 | 0.0353 | 0.0353 | yes |
| 2015-2016 | 4,809 | 0.172 | 0.778 | 0.1266 | 0.0489 | 0.0477 | 0.0489 | no |
| 2017-2018 | 4,554 | 0.193 | 0.769 | 0.1392 | 0.0569 | 0.0279 | 0.0279 | yes |
The compact view of the three policies:
Table 4. Test ECE per cycle under three maintenance policies.
| Cycle | Never | Always | Rule (recalibrated?) |
|---|---|---|---|
| 2003-2004 | 0.016 | 0.027 | 0.016 (no) |
| 2005-2006 | 0.016 | 0.023 | 0.016 (no) |
| 2007-2008 | 0.032 | 0.007 | 0.007 (yes) |
| 2009-2010 | 0.031 | 0.031 | 0.031 (yes) |
| 2011-2012 | 0.046 | 0.025 | 0.025 (yes) |
| 2013-2014 | 0.030 | 0.035 | 0.035 (yes) |
| 2015-2016 | 0.049 | 0.048 | 0.049 (no) |
| 2017-2018 | 0.057 | 0.028 | 0.028 (yes) |
| Mean | 0.034 | 0.028 | 0.026 |
Always recalibrating paid a visible price in the early cycles, where the frozen model was still well calibrated and the isotonic map added pure estimation noise (ECE 0.016 to 0.027 in 2003 to 2004): the same small sample harm quantified in our companion study, here appearing in the wild. The rule declined to recalibrate exactly there, joined the always policy once drift emerged, and achieved the best mean ECE of the three. Its mistakes are honest ones: in 2013 to 2014 it recalibrated when the holdout suggested drift that the test partition did not confirm, and in 2015 to 2016 it held at a marginal call.
Over two decades of real population change, the model's ranking ability was almost untouched while its probabilities became systematically wrong, with the miscalibration tracking the prevalence shift at a correlation of 0.96. Any monitoring program built on discrimination alone, which is the common default, would have reported this model healthy in 2017 while it understated absolute risk in the cycle with the highest prevalence in the series.
The comparison operationalizes a piece of practical guidance: recalibration is neither free (early cycles) nor optional (late cycles), and the choice can be automated with a rule that compares measured miscalibration against the known estimation noise of the correction at the available holdout size. The rule requires nothing exotic: a few hundred labelled recent cases per review period and two ECE computations.
Decomposing the miscalibration clarifies the maintenance options. A prevalence shift under stable feature outcome relationships moves calibration in the large, the overall level of predicted risk, which a one parameter intercept update can repair; changed relationships bend the calibration curve, requiring slope or nonparametric correction. The strong tracking of miscalibration by prevalence, and the fact that isotonic recalibration on a few hundred people largely repaired it, indicate the drift here is mostly of the first, benign kind: the disease became more common and the model's intercept aged. That is reassuring for maintenance, an intercept update is cheap and nearly harmless at modest sample sizes, and it sharpens the monitoring prescription: track calibration in the large first, since it is both the earliest and the cheapest failure to catch. The residual, non prevalence miscalibration in the middle cycles suggests slower secondary drifts, plausibly in adiposity distributions, that a slope term would address; the released per cycle results permit that decomposition.
Temporal and geographic transportability failures of clinical models are well documented [1, 2, 3], and periodic recalibration is standard advice [2, 8]. The contributions here are the horizon and the separation: eight consecutive evaluation cycles on a measurement stable national survey, discrimination and calibration tracked separately, and a prospective, sample size aware trigger for recalibration evaluated against both default policies over the full period.
Unweighted analysis, stated in Methods; conclusions are within sample. Five predictors were chosen for transparency, not maximal performance; a richer model might drift differently. The outcome definition mixes laboratory and self report. One survey, one disease, one country. The decision rule's noise threshold comes from a simulation calibrated to logistic base models; other model classes need their own noise curves.
Population drift breaks the meaning of a risk model's probabilities long before it breaks their order. Calibration belongs in routine model monitoring, and the decision to recalibrate can and should be made quantitatively.
All NHANES files are public. Complete source, including the XPT reader and every analysis step, regenerates the results with one command and accompanies the manuscript.
The author declares no competing interests. The work received no funding.
Figure 1. AUC and ECE of the frozen model per cycle, 2003 to 2018 (drift.svg). Figure 2. Test ECE under the three maintenance policies per cycle (strategies.svg).