Karanvir Singh
Independent researcher, Quantix Bio, Cambridge, MA, USA
Correspondence: karanvir.singh@quantixmind.com
Manuscript draft, August 2026. Not yet peer reviewed. Methods paper on simulated data; no clinical performance is claimed.
Background. Ward early warning systems are typically evaluated by discrimination metrics such as the area under the receiver operating characteristic curve, computed per observation. Yet deployed systems fail predominantly through alarm fatigue, and the quantities that determine bedside behaviour are different: how many hours of warning an alert provides before deterioration, and how many false alarms staff absorb per patient day. These quantities cannot be validated on retrospective clinical data without label noise, because true deterioration onset times are not reliably recorded.
Methods. We propose evaluating early warning systems on the operating curve of median alert lead time against false alert episodes per patient day, sweeping each system across its full threshold range, with alert episodes rather than alerting observations as the unit of burden. We demonstrate the framework on a fully synthetic ward cohort (800 patients, 72 hours of hourly vital signs, 15 percent deteriorating), where ground truth onset is known exactly by construction, comparing a NEWS2 style aggregate threshold score against logistic regression on current vitals plus four hour trends.
Results. Single threshold comparisons were uninformative or misleading: both systems detected 100 percent of events across broad threshold ranges while differing several fold in burden. On the operating curve, the trend aware model provided approximately 6.6 hours of median lead time at 0.13 false episodes per patient day, while the threshold score required roughly twice that burden (0.27 per patient day) for similar lead time. Framed in the conventional way, both systems would report near identical detection rates.
Conclusions. Lead time per unit of alarm burden makes visible the differences that determine whether an early warning system survives contact with a ward, and single threshold detection metrics hide them. We propose the operating curve as a reporting standard for early warning research, and release the simulation and evaluation code for reuse. Simulation results transfer as methodology, not as clinical performance; validating the curve on real annotated deterioration data is the necessary next step.
Keywords. early warning scores, alarm fatigue, clinical deterioration, NEWS2, evaluation methodology, patient safety
Early warning systems monitor ward patients for signs of deterioration and alert staff when risk crosses a threshold. Their evaluation literature leans on discrimination: area under the curve for predicting an event within a fixed horizon, computed per observation [1, 2]. Their failure literature tells a different story: alarm fatigue, in which the volume of false and clinically unactionable alerts trains staff to silence the system, is a recognized patient safety hazard in its own right [3, 4], and implemented scores have repeatedly shown modest or no impact on outcomes despite acceptable discrimination [5, 6]. A widely deployed proprietary sepsis model was found on external validation to combine low sensitivity at deployed thresholds with a large alerting burden [7].
The mismatch is between the metric and the mechanism. What a ward experiences is not per observation classification accuracy but a stream of alert episodes; what an alert buys is not a true positive label but time before deterioration in which to act. A system's real operating characteristics are therefore a curve: warning time gained, against alarm episodes imposed, as the threshold sweeps its range. Related arguments have been made for evaluating alerting systems by workup to detection ratios and alerts per positive case [8]; our proposal extends this to the temporal quantity, lead time, that early warning specifically promises.
Measuring lead time on retrospective clinical data is hard for a reason that is often underacknowledged: the true onset time of deterioration is not recorded, only surrogate timestamps such as ICU transfer or antibiotic administration, which embed the response of the very staff the system is meant to prompt. Simulation inverts this weakness: on synthetic cohorts the onset time is known exactly, so evaluation machinery can be developed and stress tested without label noise, then carried to real data. We present such a framework, and use it to show that two systems indistinguishable by detection rate differ two fold in the burden they impose.
We generated independent train and test cohorts of 400 simulated ward patients each, observed hourly for 72 hours across five vital signs (heart rate, respiratory rate, systolic blood pressure, oxygen saturation, temperature), with patient level baseline offsets and Gaussian observation noise. Fifteen percent of patients deteriorate: an event time is drawn uniformly between hours 24 and 72, and over the final 12 hours before it their vitals ramp toward a sepsis like pattern (rising heart rate, respiratory rate and temperature, falling blood pressure and saturation) with a smooth sub linear ramp. The generator is deliberately simple and fully specified in released code; its purpose is exact ground truth, not physiological fidelity.
Two alerting systems were evaluated. The first is a NEWS2 style aggregate score [9]: each vital is banded and assigned 0 to 3 points, points are summed, and an alert fires when the sum crosses a threshold, swept over 3 to 9. The second is logistic regression on the five current vitals plus their four hour slopes, trained on the training cohort to predict deterioration within the next six hours, with its probability threshold swept over 0.05 to 0.70. The comparison is deliberately between a representative threshold score and a minimal learned model using trend information the former discards; neither is tuned beyond the description above.
For each system and threshold, on the held out cohort: median lead time, the time from the first alert preceding the event to the event, zero if no alert precedes it; detection fraction, the proportion of deteriorating patients with any pre event alert; and false alert episodes per patient day among non deteriorating patients, where an episode is a rising edge, a transition from non alerting to alerting state, so that a threshold sat crossed for twelve consecutive hours counts once, matching how staff experience an alarm rather than how observations are labelled.
The entire framework is implemented in numpy, runs in seconds, and is released with the manuscript. All numbers below regenerate from one command.
Across threshold ranges spanning a fourteen fold change in alarm burden, both systems detected 100 percent of deteriorating patients. A single threshold comparison of detection or sensitivity would report the systems as equivalent, or rank them arbitrarily depending on the thresholds chosen.
Table 1. Selected operating points, held out cohort.
| System | Threshold | Median lead time (h) | Detection | False episodes per patient day |
|---|---|---|---|---|
| NEWS2 style | 3 | 24.6 | 100% | 1.17 |
| NEWS2 style | 4 | 6.6 | 100% | 0.27 |
| NEWS2 style | 5 | 5.1 | 100% | 0.045 |
| Trend model | 0.05 | 13.6 | 100% | 0.38 |
| Trend model | 0.10 | 6.9 | 100% | 0.20 |
| Trend model | 0.15 | 6.6 | 100% | 0.13 |
| Trend model | 0.25 | 5.7 | 100% | 0.092 |
At matched lead time of about 6.6 hours, the trend aware model imposed roughly half the alarm burden of the threshold score (0.13 versus 0.27 false episodes per patient day). Read the other way, at matched burden near 0.27 per patient day the threshold score bought 6.6 hours of median warning while the trend model at comparable burden (0.20) bought similar warning with headroom to spare. The threshold score achieves very low burden only at its highest thresholds, where median lead time contracts toward 2 to 5 hours. The full curves are shown in Figure 1.
Counting alert episodes rather than alerting observations changed the ranking of operating points in preliminary runs: a score that flickers across its threshold generates many episodes from few alerting hours, and episodes are what interrupt a nurse. Similarly, expressing burden per patient day rather than per observation ties the number to staffing reality: 0.27 false episodes per patient day on a 30 bed ward is roughly eight interruptions per day; 0.13 is four. The units are the argument.
Table 2. Every operating point evaluated, held out cohort.
| System | Threshold | Median lead (h) | Detection | False episodes per patient day |
|---|---|---|---|---|
| NEWS2 style | 3 | 24.6 | 100% | 1.168 |
| NEWS2 style | 4 | 6.6 | 100% | 0.274 |
| NEWS2 style | 5 | 5.1 | 100% | 0.045 |
| NEWS2 style | 6 | 4.6 | 100% | 0.006 |
| NEWS2 style | 7 | 3.7 | 100% | 0.000 |
| NEWS2 style | 8 | 2.8 | 100% | 0.000 |
| NEWS2 style | 9 | 2.3 | 100% | 0.000 |
| Trend model | 0.05 | 13.6 | 100% | 0.380 |
| Trend model | 0.10 | 6.9 | 100% | 0.197 |
| Trend model | 0.15 | 6.6 | 100% | 0.134 |
| Trend model | 0.20 | 6.4 | 100% | 0.107 |
| Trend model | 0.25 | 5.7 | 100% | 0.092 |
| Trend model | 0.30 | 5.6 | 100% | 0.077 |
| Trend model | 0.35 | 5.3 | 100% | 0.072 |
| Trend model | 0.40 | 4.8 | 100% | 0.070 |
| Trend model | 0.45 | 4.8 | 100% | 0.063 |
| Trend model | 0.50 | 4.2 | 98% | 0.056 |
| Trend model | 0.55 | 4.2 | 98% | 0.054 |
| Trend model | 0.60 | 4.0 | 96% | 0.052 |
| Trend model | 0.65 | 3.5 | 96% | 0.051 |
| Trend model | 0.70 | 2.6 | 93% | 0.050 |
The full sweep makes the single threshold problem concrete: a reader comparing "NEWS2 at threshold 5" against "model at 0.25" would conclude near equivalence, while comparing threshold 4 against 0.15 shows the model at half the burden. Neither comparison is wrong; both are arbitrary. Only the curve is a property of the system rather than of a reporting choice.
The contribution is the evaluation frame, demonstrated where it can be measured without label noise. Two systems that a conventional evaluation would call equivalent differ two fold in the burden they impose for the same warning. Alarm fatigue is the documented failure mode of deployed systems [3, 4, 5, 7]; an evaluation that cannot see burden per unit of benefit cannot predict that failure.
The synthetic cohort establishes the machinery: definitions, units, and code that measure lead time exactly when ground truth exists. It does not establish that any particular model beats NEWS2 clinically. The generator's deterioration is a smooth monotone ramp, which plausibly flatters trend features; real deterioration includes abrupt collapses, intermittent measurement, treatment responses, and missingness correlated with concern. The stated numbers are properties of the simulator, offered to demonstrate the method.
The framework transfers to real cohorts with annotated deterioration events, accepting that onset timestamps there are surrogates. A natural path is dual evaluation: report the operating curve against both a clinical surrogate (ICU transfer, rapid response call) and a sensitivity analysis over plausible onset offsets. This program has since been carried out: a companion study applies this framework unchanged to hourly vitals from 4,755 real ICU patients across two hospital systems, with derivation on one and external validation on the other, and finds that the framework's verdicts sharpen rather than soften on real data. The present paper stands as the methodology reference: the simulation setting remains the only one in which lead time is measurable without label noise, and therefore the right place to define and stress test the machinery.
Reports of early warning systems should include, alongside discrimination: the full operating curve of median lead time against false alert episodes per patient day; the episode definition used; and the operating point proposed for deployment with its projected alarms per ward per day. None of this requires new data collection, only different accounting.
All data in this study are generated by the released code; no patient data of any kind was used. Complete source, including the cohort generator, both scoring systems and the evaluation harness, is available from the author and will be archived with the preprint.
The author declares no competing interests. The work received no funding.
Figure 1. Median lead time versus false alert episodes per patient day for both systems across their threshold ranges (tradeoff.svg). Figure 2. Detection fraction versus alarm burden (detection.svg).