Karanvir Singh
Independent researcher, Quantix Bio, Cambridge, MA, USA
Correspondence: karanvir.singh@quantixmind.com
Manuscript draft, August 2026. Not yet peer reviewed.
Background. Selective prediction lets a clinical risk model abstain on its least confident cases and defer them to a clinician, and is increasingly proposed as a safety mechanism for clinical machine learning. Its value depends on an empirical premise that is usually assumed rather than measured: that the model's errors concentrate in its low confidence region. We measured how strongly that premise holds across a range of clinical prediction tasks, and how much accuracy abstention actually buys at fixed deferral budgets.
Methods. On eight public biomedical classification datasets (n = 303 to 768), we trained L2 regularized logistic regression with five fold cross validation, scored confidence as distance of the out of fold predicted probability from 0.5, and computed coverage versus selective accuracy curves over 20 repeated cross validation runs. We report accuracy at 100, 80 and 50 percent coverage, the share of all errors captured in the least confident 20 and 30 percent of cases, and the area under the selective error versus coverage curve. Two further uncertainty estimators were run to test whether richer uncertainty rescues concentration on hard tasks: a bagged ensemble of 15 logistic models with confidence from the mean probability, and split conformal prediction with class conditional calibration, confidence scored as the conformal p value margin. All methods are implemented from first principles in numpy; everything regenerates from public data with one command.
Results. Error concentration varied widely and split the tasks into two regimes. On the two high separability tasks (breast mass diagnosis by two feature sets; full coverage accuracy 0.97 to 0.98), the least confident 20 percent of cases contained 88 to 90 percent of all errors, and abstaining on that fifth raised selective accuracy to 0.995 or better, essentially eliminating error at 20 percent deferral. On the six harder tasks (full coverage accuracy 0.72 to 0.83), the least confident fifth contained only 32 to 44 percent of errors, between 1.6 and 2.2 times the uninformative baseline of 20 percent, and reaching high selective accuracy required deferring half of all cases (accuracy at 50 percent coverage 0.82 to 0.93). Ordering datasets by full coverage accuracy ordered their error concentration almost perfectly.
Neither alternative rescued the hard tasks. The bagged ensemble left error concentration essentially unchanged everywhere (largest change 0.02). Conformal confidence performed no better, and on several hard datasets slightly worse, than the raw probability margin (liver 0.29 versus 0.32, donation 0.28 versus 0.42, survival 0.25 versus 0.35), while matching it on the separable tasks (0.94 and 0.82). Validity guarantees order coverage, not error location.
Conclusions. Abstention is not a uniform safety mechanism. Where a task is nearly separable, confidence based deferral of one case in five removes nearly all errors, a striking and deployable property. Where the task is genuinely hard, the same mechanism removes only a third of errors at the same budget, because errors arise in regions the model does not know it is wrong about. Selective prediction proposals for clinical use should report error concentration, not only coverage accuracy curves, and should not extrapolate the near separable behaviour to hard tasks.
Keywords. selective prediction, abstention, reject option, clinical decision support, uncertainty, human AI collaboration
A model that can say "I am not sure, you decide" is an attractive shape for clinical machine learning: the machine handles clear cases, the ambiguous ones route to a human, and stated performance applies only to what the machine actually answers. The idea is old, beginning with Chow's optimal rejection rule [1], and has an established modern literature as selective prediction or learning with a reject option [2, 3, 4], with growing interest in clinical applications where deferral to a clinician is the natural fallback [5, 6].
The mechanism only works to the extent that a model's errors live where its confidence is low. That premise is routinely illustrated on benchmark image datasets with high base accuracy [3, 4], a regime where most predictions are correct and confident, and errors have little room to hide. Clinical tabular prediction commonly lives in a different regime: base accuracies of 0.70 to 0.85, substantial irreducible uncertainty, and, as a consequence, a risk that errors are distributed across the confidence range rather than concentrated at the bottom.
This study measures the premise directly. Across eight public clinical classification datasets spanning both regimes, we quantify how much of the total error mass sits in the least confident cases, how much accuracy abstention buys at realistic deferral budgets, and how these quantities relate to base task difficulty.
The eight datasets are those of our companion calibration study, spanning coronary disease (two cohorts), breast mass diagnosis (two feature sets), diabetes onset, liver disease, blood donation return and five year cancer survival (Table 1 of that study; n = 303 to 768, prevalence 0.24 to 0.71) [7 to 12]. The model was L2 regularized logistic regression (lambda = 1.0 on standardized features) fit by Newton's method. Predictions were generated out of fold under five fold cross validation, so no case is scored by a model that saw it. Confidence was the distance of the predicted probability from 0.5, the natural score for a binary probabilistic classifier and equivalent to ranking by maximum class probability.
For each dataset we computed, over 20 repeated cross validation partitions: selective accuracy at coverages 1.0 to 0.5 in steps of 0.1, where coverage c means answering the most confident fraction c of cases; the error concentration at 20 and 30 percent, defined as the share of all errors falling in the least confident 20 or 30 percent of cases (an uninformative confidence score gives 0.20 and 0.30; an oracle gives 1.0 whenever the error rate is below the abstention budget); and the area under the selective error versus coverage curve from coverage 0.1 to 1.0, normalized by the coverage range.
All machinery is implemented in numpy from first principles and runs from raw public data in seconds. Code and per cell results accompany the manuscript.
Table 1. Selective prediction across eight datasets (means over 20 repeated cross validations).
| Dataset | Acc at 100% | Acc at 80% | Acc at 50% | Errors in least confident 20% | Errors in least confident 30% |
|---|---|---|---|---|---|
| WDBC (breast, morphometry) | 0.978 | 0.997 | 0.999 | 0.90 | 0.91 |
| Wisconsin original (breast, cytology) | 0.966 | 0.995 | 0.997 | 0.88 | 0.92 |
| Cleveland (coronary) | 0.831 | 0.881 | 0.931 | 0.44 | 0.59 |
| Pima (diabetes) | 0.773 | 0.822 | 0.895 | 0.37 | 0.53 |
| Transfusion (donation) | 0.771 | 0.834 | 0.892 | 0.42 | 0.56 |
| Haberman (survival) | 0.744 | 0.791 | 0.816 | 0.35 | 0.46 |
| ILPD (liver) | 0.718 | 0.761 | 0.863 | 0.32 | 0.47 |
| SA heart (coronary) | 0.723 | 0.769 | 0.843 | 0.33 | 0.47 |
On the two near separable tasks, the least confident fifth of cases held 88 to 90 percent of all errors. Abstaining on that fifth raised accuracy on answered cases from 0.97 to 0.995 or better: at a 20 percent deferral budget the machine's error was essentially eliminated. On the six harder tasks the same budget captured only 32 to 44 percent of errors, and selective accuracy at 80 percent coverage improved by only 4 to 6 points. Even at 50 percent coverage, half of all cases deferred, selective accuracy on the hard tasks reached only 0.82 to 0.93.
Ranking the eight datasets by full coverage accuracy reproduced their ranking by error concentration almost exactly. This is what theory leads one to expect: near the separable regime, errors are rare borderline cases that sit at low confidence by construction, while on hard tasks a large share of error is irreducible class overlap distributed across the confidence range, including regions where the model is confidently wrong. The empirical contribution is the magnitude of the gap: a factor of about 4.5 over the uninformative baseline on separable tasks versus 1.6 to 2.2 on hard ones, measured under an identical protocol.
A bagged ensemble of 15 logistic models, whose averaged probabilities reduce estimation variance in the confidence score, changed error concentration by at most 0.02 on any dataset (for example liver 0.32 to 0.33, breast morphometry 0.90 to 0.92). Split conformal prediction with class conditional calibration, whose confidence carries finite sample validity guarantees, did no better: error concentration in the least confident fifth was 0.44 (coronary), 0.35 (diabetes), 0.29 (liver), 0.28 (donation) and 0.25 (survival), equal to or below the raw probability margin on every hard task, while matching it on the separable ones (0.94, 0.82). The pattern supports a single interpretation: on hard tasks the residual error is dominated by irreducible class overlap, which neither variance reduction nor validity guarantees can relocate into the low confidence region, because the model's confidence is not wrong about where it is uncertain; the task simply is uncertain in places where the model is confident and correct to be so on average.
Table 2. Share of all errors in the least confident 20 percent of cases, three confidence scores.
| Dataset | Single model | Bagged ensemble | Conformal margin |
|---|---|---|---|
| cleveland | 0.44 | 0.44 | 0.44 |
| wdbc | 0.90 | 0.92 | 0.94 |
| pima | 0.37 | 0.38 | 0.35 |
| transfusion | 0.42 | 0.41 | 0.28 |
| haberman | 0.35 | 0.34 | 0.25 |
| wisc_original | 0.88 | 0.90 | 0.82 |
| ilpd | 0.32 | 0.33 | 0.29 |
| saheart | 0.33 | 0.33 | 0.31 |
Table 3. Full coverage accuracy grid and area under the selective error coverage curve (AURC), single model confidence.
| Dataset | 100% | 90% | 80% | 70% | 60% | 50% | AURC |
|---|---|---|---|---|---|---|---|
| cleveland | 0.831 | 0.857 | 0.881 | 0.902 | 0.918 | 0.931 | 0.0802 |
| wdbc | 0.978 | 0.992 | 0.997 | 0.997 | 0.997 | 0.999 | 0.0028 |
| pima | 0.773 | 0.794 | 0.822 | 0.847 | 0.869 | 0.895 | 0.1265 |
| transfusion | 0.771 | 0.809 | 0.834 | 0.855 | 0.878 | 0.892 | 0.1261 |
| haberman | 0.744 | 0.768 | 0.791 | 0.803 | 0.812 | 0.816 | 0.2112 |
| wisc_original | 0.966 | 0.991 | 0.995 | 0.996 | 0.997 | 0.997 | 0.0046 |
| ilpd | 0.718 | 0.738 | 0.761 | 0.785 | 0.821 | 0.863 | 0.1473 |
| saheart | 0.723 | 0.746 | 0.769 | 0.792 | 0.819 | 0.843 | 0.1675 |
The three scores agree to within a few points on every dataset, and the agreement is the finding: the ranking of cases by uncertainty is essentially fixed by the task, and the choice among reasonable uncertainty estimators moves error concentration by amounts that are negligible next to the gap between the two regimes.
Expressed as a clinical planning quantity: at a fixed budget of one case in five sent to a human, a breast mass classifier on these features removes about nine tenths of its errors, while a diabetes, liver or survival model removes about a third. For the harder tasks, the coverage accuracy curves (Figure 1) are close to linear, meaning each additional deferred case buys roughly constant, modest error reduction: there is no elbow after which the machine's remaining answers become trustworthy.
The safety story of selective prediction, machine handles the clear cases and defers the rest, held quantitatively in only one of the two regimes studied. Its strong form is real and striking on nearly separable tasks. On the harder tasks typical of tabular clinical risk prediction, confidence based abstention helped, always beating the uninformative baseline, but far too weakly to serve as a safety guarantee: most errors survived any realistic deferral budget.
First, proposals that quote selective prediction as a safety mechanism should report error concentration at stated budgets, not just coverage accuracy curves, because the latter look monotonically reassuring even when concentration is weak. Second, the base accuracy of a task is a cheap and evidently strong predictor of whether abstention will function as intended; a system designer can measure it before promising deferral behaviour. Third, on hard tasks the residual errors are by definition confident errors, which are the clinically dangerous kind; abstention shifts the error mix toward them, a side effect deployment analyses should state.
One model class was studied, with three confidence scores: the single model's probability, a bagged ensemble mean, and class conditional split conformal margins, all of which concentrated error equivalently. Richer aleatoric models and learned deferral policies [6] remain the open direction. The datasets are modest sized public research cohorts. Confidence was not recalibrated before thresholding, deliberately, since ranking by distance from 0.5 is invariant to monotone recalibration; our companion study addresses recalibration separately. Coverage accuracy analysis treats all errors equally, while clinical costs are asymmetric; a cost weighted extension is straightforward within the released code.
Abstention is a property of the task before it is a property of the model. Reporting error concentration alongside coverage makes the difference visible, and would prevent the strongest advertised behaviour of selective prediction, measured on separable benchmarks, from being implicitly promised on the hard tasks where clinical models actually live.
All datasets are public and cited in the companion study. Complete numpy source, per dataset results and figures regenerate with one command and accompany the manuscript.
The author declares no competing interests. The work received no funding.
Figure 1. Selective accuracy gain over full coverage versus coverage, eight datasets (gain_vs_coverage.svg). Figure 2. Share of all errors captured in the least confident 20 percent of cases per dataset, against the uninformative baseline of 0.20 (error_concentration.svg).