MATHEMATICAL MODELS TO ESTIMATE THE TRUE (ADDED) DIAGNOSTIC ACCURACY OF A TEST

Dr Feinstein argues that diagnostic research is currently unfortunately dictated by mathematical, quantitative models, often ignoring the practitioners' judgement. We, however, believe that these quantitative models are necessary in order to estimate the true, independent (or added) value of a test, just as they are necessary in aetiological studies to estimate the independent association of a particular exposure and the occurrence of a particular outcome.

In medical practice, the diagnostic investigation starts with a patient presenting with a particular symptom or sign indicative for the presence of a particular disease, the so called target disease.² The diagnostic investigation is a consecutive (hierarchical) process always starting from patient history and physical examination, followed by more invasive, time consuming and costly tests such as imaging. It amounts to the estimation of the probability of presence of the target disease given all test results, further referred to as the diagnostic probability.³ In practice no diagnosis is set by a single test result and each test result is judged together with other (previous) test results. Given this consecutive investigation, the fact that different tests provide to varying extents the same information and that each test may be more or less burdening for the patient, time consuming and costly, physicians need to know which tests additionally or independently contribute to the diagnostic probability estimation.^1–11 For example, to determine whether an exercise stress test in patients suspected of coronary artery disease has diagnostic value, one needs to quantify whether the test changes the diagnostic probability based on previous test results (for example, patient history and physical examination) that are commonly available anyway before such test is applied.

In our view, to quantify whether a particular test result additionally increases or decreases the diagnostic probability requires multivariable (prediction) modelling of the occurrence (prevalence) of the disease at issue as function of the different diagnostic test results. It requires a comparison of the probability (prevalence) of disease presence before and after the test, using, for example, stepwise logistic regression modelling according to the chronology in practice. The result of such analyses is the definition of one or more diagnostic prediction models including the relevant (contributing) tests. If needed, these models can be simplified to obtain easy applicable diagnostic decision rules for use in practice.^12–15

Knowledge on the independent (added) diagnostic value can, as Dr Feinstein also suggests, not be inferred from the still widely used singular test parameters—that is, a test's sensitivity, specificity, and likelihood ratio.^1,11 Similarly to the variability of these parameters across patient characteristics, the independent value of a test can also not be validly quantified by the use of Bayes's theorem.³ The need for multivariable models in diagnostic research is not different form other types of medical research such as aetiological and prognostic research. Not the singular association (odds ratio) between a particular exposure and the occurrence of the outcome is informative, but their association independent of other (co) factors.

PRACTITIONERS' JUDGEMENT TO ASSESS DIAGNOSTIC ACCURACY

Dr Feinstein argues that physicians are generally reluctant to use quantitative prediction models in their diagnostic practice but rather infer a test's accuracy from their own experience. In our view, this should be no reason to withdraw from the use of quantitative prediction models in diagnostic research.

For the evaluation of efficacy of preventive and therapeutic devices we have left the era of “experience or eminence based medicine” in favour of “evidence based medicine”. This should also apply to diagnostic technologies.

Resistance to use prediction models in diagnostic practice is most likely because they are still seldomly used and reported in diagnostic studies, in contrast with aetiological and prognostic, including therapeutic, research. For the latter types of research a proper methodological framework encouraging the use of these methods can be found in all epidemiological textbooks. This is not the case for the few textbooks addressing methods of diagnostic research. Most diagnostic research still concentrates on the estimation of singular test parameters in the analyses. To enable diagnostic research to better serve practice the prevailing framework of methods for both design and analyses must be improved.

We agree, however, that the physician's judgement or experience is often an important “test” in the diagnostic investigation, which should not be ignored in research. However, we believe it is preferable to express this experience in more or less objective test results, or at least to quantify whether this “experience test” provides added information to the more objective test results (as described above).

THE VALUE OF TEST RESULTS IN TERMS OF PATIENT OUTCOME

To set a diagnosis is fundamental in medical care as it offers an indication of the patient's prognosis and directs therapeutic management. As Dr Feinstein suggests, a diagnostic technology may not contribute to the assessment of the final diagnosis, but rather provide information that could be relevant to the patient's prognosis or to therapeutic decisions.

We agree that diagnostic technologies should not only be evaluated on their diagnostic accuracy, for example, their ability to determine the presence or absence of the disease but sometimes also on their ability to change patient outcome.^16–19 These assessments, however, call for different types of outcomes and thus different study designs. Evaluation of a test on its (added) diagnostic accuracy requires a multivariable, cross sectional prediction study: of each subject the result of the test under study is measured as well as the simultaneous presence or absence of the disease as determined by a reference. Evaluating a test on patient outcome comprises the evaluation of diagnostic tests plus all possible administered therapies combined. This enters the realm of aetiological and therapeutic research and requires follow up studies or trials rather than cross sectional, prediction studies.^19,20 We believe that evaluation of diagnostic tests on patient outcome is not always necessary. In general, we think that follow up studies are not necessary and the (beneficial) effect of a diagnostic test for patient outcome may be considered as established if (1) diagnostic (cross sectional) studies have shown the test's ability to detect a particular disease and (2) therapeutic studies provided evidence on efficacy of the management of this disease.²⁰ Follow up studies to quantify the effect of a diagnostic technology on patient outcome are necessary (1) if the disease at issue lacks a specific reference to determine its presence or absence (such as heart failure) or (2) if the (new) diagnostic technology provides other therapeutic information than the reference, potentially leading to other treatment choices or (3) if the technology itself may have therapeutic properties such as salphingography to determine patency of the uteral tubes.