Nondifferential disease misclassification may bias incidence risk ratios away from the null

doi:10.1016/j.jclinepi.2005.07.013

Journal of Clinical Epidemiology

Volume 59, Issue 3, March 2006, Pages 281-289

https://doi.org/10.1016/j.jclinepi.2005.07.013 Get rights and content

Abstract

Background and Objective

When estimating incidence risk ratios in follow-up studies, subjects testing positive for the disease at baseline are excluded. Although the effect of disease misclassification on estimated incidence risk ratios has otherwise been extensively explored, the effect of disease misclassification at baseline has not previously been analyzed.

Study Design and Setting

The design was theoretical calculations assuming dichotomous disease and a follow-up study with a baseline and a follow-up examination, analyzed using cumulative incidence. Calculations consider nondifferential misclassification of disease mainly at baseline, but no misclassification of exposure.

Results

Nondifferential misclassification of disease at baseline can lead to bias either away or toward null in estimated cumulative incidence risk ratios. This bias is mainly a function of sensitivity at baseline, because imperfect sensitivity leads to failure to exclude all diseased subjects from the follow-up. Imperfect specificity at baseline has less effect. Bias is increased with high true prevalence of disease and low true incidence. Bias is also increased with large differences in true risk ratios at baseline and at follow-up, because observed incidence risk ratios in the presence of misclassification reflect both the true association at baseline and at follow-up.

Conclusion

Nondifferential disease misclassification at baseline examination of a follow-up study can lead to over- or underestimation of the cumulative incidence risk ratios. The bias can be substantial for disease with low incidence and high prevalence, such as asthma or myocardial infarction. The results underscore the need to select a highly sensitive test for disease at baseline to exclude all diseased subjects from the follow-up.

Introduction

An analysis of incidence is the standard method to analyze the association between baseline exposure and risk of disease in a follow-up study. Cumulative incidence among both exposed and nonexposed is calculated as number of new cases of disease among the population at risk (i.e., for chronic diseases, subjects free of disease at the beginning of follow-up). Excluding subjects with prevalent disease from the follow-up is meant to ensure that the study can assess causality truly prospectively—that is, measured exposure precedes onset of disease. When calculating incidence, it is commonly assumed that there is no error in excluding prevalent cases of disease from the follow-up.

In a follow-up study, nondifferential misclassification of disease during follow-up leads in general to bias toward null in the estimated relative risks [1]. This bias is mainly a function of specificity [1], especially for rare diseases, and in the presence of perfect specificity there is no bias [2].

Effect of misclassification of disease at baseline has rarely been considered. Recently it was suggested that nondifferential misclassification of binary disease can lead to bias either away or toward null in odds ratios for transition probabilities in a follow-up study [3]. In a follow-up study, it was shown that measurement error in continuous outcomes can lead to bias either away or toward null when the analysis is adjusted for the baseline level of the outcome [4], [5]. It has also recently been shown that measurement error can severely bias estimates of incidence of asthma (unpublished data, 2003). To our knowledge, however, the effect of disease misclassification at baseline on the estimated cumulative incidence risk ratios has not previously been analyzed.

Our objective here was to explore the magnitude and determinants of bias in the observed cumulative incidence risk ratios in a follow-up study of a dichotomous disease measured with error at baseline.

Section snippets

Effect of misclassification of disease on estimated incidence

We first illustrate the effect of misclassification of disease on the estimated cumulative incidence (Fig. 1). For added clarity, all possible 16 combinations of true and observed changes in disease status in an analysis of cumulative incidence are also shown in Table 1. Note that we consider only cumulative incidence, assume that misclassification of disease is nondifferential, and do not consider misclassification of exposure. We also assume that risk of remission of disease is independent of

Discussion

Our present results show that nondifferential misclassification of disease at baseline, especially imperfect sensitivity, can lead to bias away or toward null in the observed cumulative incidence risk ratios. This is in contrast to nondifferential misclassification of disease during follow-up, which in general leads to bias toward null [1]. The bias described can be substantial for disease with low incidence and high prevalence, such as asthma or myocardial infarction. The results underscore

Acknowledgments

We thank Pia Verkasalo, MD, for helpful comments on the draft manuscript.

References (23)

E.S. Ford et al.
Prevalence of nonfatal coronary heart disease among American adults
Am Heart J
(2000)
K. Kuulasmaa et al.
Estimation of contribution of changes in classic risk factors to trends in coronary-event rates across the WHO MONICA Project populations
Lancet
(2000)
K.T. Copeland et al.
Bias due to misclassification in the estimation of relative risk
Am J Epidemiol
(1977)
C. Poole
Exception to the rule about nondifferential misclassification
Am J Epidemiol
(1985)
R. Wolfe et al.
Transitions in an imperfectly observed binary variable: depressive symptomatology in adolescents
Stat Med
(2003)
L.E. Chambless et al.
Methods for assessing difference between groups in change when initial measurements is subject to intra-individual variation
Stat Med
(1993)
N.D. Yanez et al.
The effects of measurement error in response variables and tests of association of explanatory variables in change models
Stat Med
(1998)
W. Buse et al.
Childhood versus adulthood onset asthma
Am J Respir Crit Care Med
(1995)
D. Jarvis et al.
European Community Respiratory Health Survey II Steering Committee. The European Community Respiratory Health Survey II
Eur Respir J
(2002)
M.A. Jenkins et al.
Validation of questionnaire and bronchial hyperresponsiveness against respiratory physician assessment in the diagnosis of asthma
Int J Epidemiol
(1996)

C.M. Albert et al.

Triggering of sudden death from cardiac causes by vigorous exertion

N Engl J Med

(2000)

Cited by (25)

Randomized clinical trials and observational studies in the assessment of drug safety
2018, Revue d'Epidemiologie et de Sante Publique
Citation Excerpt :
These different biases can distort the interpretation of OS and result in misleading findings [61–66]. Note also that non-differential misclassification does not always result in bias towards the null, as it is sometimes assumed [67,68]. An important observation is that indication and selection bias would theoretically have limited impact when evaluating the risk for unintended or unanticipated effects like unexpected ADRs because the assignment to the treatment is not expected to be affected by these factors [19,69,70].
Randomized clinical trials are considered as the preferred design to assess the potential causal relationships between drugs or other medical interventions and intended effects. For this reason, randomized clinical trials are generally the basis of development programs in the life cycle of drugs and the cornerstone of evidence-based medicine. Instead, randomized clinical trials are not the design of choice for the detection and assessment of rare, delayed and/or unexpected effects related to drug safety. Moreover, the highly homogeneous populations resulting from restrictive eligibility criteria make randomized clinical trials inappropriate to describe comprehensively the safety profile of drugs. In that context, observational studies have a key added value when evaluating the benefit-risk balance of the drugs. However, observational studies are more prone to bias than randomized clinical trials and they have to be designed, conducted and reported judiciously. In this article, we discuss the strengths and limitations of randomized clinical trials and of observational studies, more particularly regarding their contribution to the knowledge of medicines’ safety profile. In addition, we present general recommendations for the sensible use of observational data.
Les essais cliniques randomisés sont considérés comme le modèle préféré pour évaluer les relations causales potentielles entre médicaments ou d’autres interventions médicales et les effets attendus. Pour cette raison, les essais cliniques randomisés sont généralement la base des programmes de développement dans le cycle de vie des médicaments et la pierre angulaire de la médecine fondée sur les preuves. Cependant, les essais cliniques randomisés ne constituent pas le modèle de choix pour la détection et l’évaluation des effets inattendus liés à la sécurité des médicaments. De plus, les populations très homogènes résultant de critères d’éligibilité restrictifs rendent les essais cliniques randomisés inappropriés pour décrire de façon exhaustive le profil de sécurité des médicaments. Dans ce contexte, les études observationnelles ont une valeur ajoutée majeure lors de l’évaluation du rapport bénéfice-risque des médicaments. Cependant, les études observationnelles sont plus sujettes aux biais que les essais cliniques randomisés et elles doivent être conçues, menées et rapportées très judicieusement. Dans cet article, nous discutons les points forts et les limites des essais cliniques randomisés et des études observationnelles, plus particulièrement eu égard à leur contribution à la connaissance du profil de sécurité des médicaments. En outre, nous exposons des recommandations pour une utilisation rationnelle des données observationnelles.
Diagnosing intramammary infection: Controlling misclassification bias in longitudinal udder health studies
2018, Preventive Veterinary Medicine
Citation Excerpt :
The effect of misclassification of disease at baseline to calculate incidence has less frequently been considered. In longitudinal studies, the nondifferential misclassification of disease at baseline, especially imperfect sensitivity, can lead to over- or under-estimation of the observed cumulative incidence risk ratios (Pekkanen et al., 2006). This bias can be significant for disease with a low true incidence, a high true prevalence, a substantial disease duration (i.e. as long as the interval between first and second test), and a poor test Se.
Using imperfect tests may lead to biased estimates of disease frequency and of associations between risk factors and disease. For instance in longitudinal udder health studies, both quarters at risk and incident intramammary infections (IMI) can be wrongly identified, resulting in selection and misclassification bias, respectively. Diagnostic accuracy can possibly be improved by using duplicate or triplicate samples for identifying quarters at risk and, subsequently, incident IMI.
The objectives of this study were to evaluate the relative impact of selection and misclassification biases resulting from IMI misclassification on measures of disease frequency (incidence) and of association with hypothetical exposures. The effect of improving the sampling strategy by collecting duplicate or triplicate samples at first or second sampling was also assessed.
Data sets from a hypothetical cohort study were simulated and analyzed based on a separate scenario for two common mastitis pathogens representing two distinct prevailing patterns. Staphylococcus aureus, a relatively uncommon pathogen with a low incidence, is identified with excellent sensitivity and almost perfect specificity. Coagulase negative staphylococci (CNS) are more prevalent, with a high incidence, and with milk bacteriological culture having fair Se but excellent Sp. The generated data sets for each scenario were emulating a longitudinal cohort study with two milk samples collected one month apart from each quarter of a random sample of 30 cows/herd, from 100 herds, with a herd-level exposure having a known strength of association. Incidence of IMI and measure of association with exposure (odds ratio; OR) were estimated using Markov Chain Monte Carlo (MCMC) for each data set and using different sampling strategies (single, duplicate, triplicate samples with series or parallel interpretation) for identifying quarters at risk and incident IMI.
For S. aureus biases were small with an observed incidence of 0.29 versus a true incidence of 0.25 IMI/100 quarter-month. In the CNS scenario, diagnostic errors in the two samples led to important selection (40 IMI/100 quarter-month) and misclassification (23 IMI/100 quarter-month) biases for estimation of IMI incidence, respectively. These biases were in opposite direction and therefore the incidence measure obtained using single sampling on both the first and second test (29 IMI/100 quarter-month) was exactly the true value.
In the S. aureus scenario the OR for association with exposure showed little bias (observed OR of 3.1 versus true OR of 3.2). The CNS scenario revealed the presence of a large misclassification bias moving the association towards the null value (OR of 1.7 versus true OR of 2.6). Little improvement could be brought using different sampling strategies aiming at improving Se and/or Sp on first and/or second sampling or using a two out of three interpretation for IMI definition.
Increasing number of samples or tests can prevent bias in some situations but efforts can be spared by holding to a single sampling approach in others. When designing longitudinal studies, evaluating potential biases and best sampling strategy is as critical as the choice of test.
The fluctuation of nocturia in men with lower urinary tract symptoms allocated to placebo during a 12-month randomized, controlled trial
2014, Journal of Urology
Citation Excerpt :
Using these questionnaires individuals may report their estimated average nocturia frequency only in whole numbers (0 to 5 or greater). Because nocturia frequency varies from night to night and self-reports are an estimated average during a given period, there is imprecision and variability in any given nocturia report, which may bias incidence estimates in either direction.10 Our results suggest that multiple questionnaire based assessments likely yield more accurate estimates of the true incidence in the setting of a prevalent condition such as nocturia, of which the long-term clinically relevant incidence may be small compared to short-term fluctuation.
We determined the fluctuation of nocturia in a 12-month period in men with lower urinary tract symptoms.
Men with lower urinary tract symptoms were allocated to the placebo arm of the United States Department of Veterans Affairs Cooperative Studies Program Benign Prostatic Hyperplasia Study. Reported nocturia frequency using the American Urological Association Symptom Index was collected at 6 time points (2, 4, 13, 26, 39 and 52 weeks). Repeat measurements of nocturia during a 1-year period were analyzed using a generalized mixed linear model.
Of the 305 men allocated to the placebo group 256 participants (84%) gave answers for all 6 time points. In the entire sample the mean nocturia count did not significantly vary from baseline (week 2) after adjusting for covariates (p = 0.542). However, there was considerable fluctuation in nocturia during 1 year. Of the 93 men with 3 or 4 episodes at baseline 47% had improvement and 12% had worsening at 1 year. Of the 184 men who reported 2 or greater nocturia episodes at baseline 15% reported 0 or 1 at 52 weeks. Depending on the case definition during followup the probability of nocturia progression varied between 8% and 54% while nocturia regression varied between 2% and 33%.
Using repeat questionnaire based assessments we observed considerable fluctuation in nocturia. However, overall there was no significant increase in prevalence in a 1-year period. These findings may be reassuring to providers and patients who elect to delay interventions for nocturia.
Accuracy of national mortality codes in identifying adjudicated cardiovascular deaths
2011, Australian and New Zealand Journal of Public Health
This study investigated the sensitivity and specificity of the national mortality codes in identifying cardiovascular disease (CVD) deaths and documents methods of verification.
A 12‐year retrospective case ascertainment of all ICD‐coded CVD deaths was performed for deaths between 1990 and 2002 in the Melbourne Collaborative Cohort Study, comprising 41,528 subjects. Categories of non‐CVD codes were also examined. Stratified samples of 750 deaths were adjudicated from a total of 2,230 deaths. Expert panels of cardiologists and neurologists adjudicated deaths.
Of the 750 deaths adjudicated, 582 were verified as CVD [392 coronary heart disease (CHD) and 92 stroke] and 168 non‐CVD. Estimated sensitivity and specificity of national mortality codes for identifying specific causes of death were: CHD 74.2% (95% CI: 69.8–78.5%) and 97.6% (96.0–99.2%), respectively; myocardial infarction 59.9% (50.9–69.0%) and 94.2% (92.4–96.0%), respectively; haemorrhagic stroke 58.9% (46.0–71.7%) and 99.8% (99.4–100.0%), respectively and; ischaemic stroke 38.7% (20.5–56.9%) and 99.9% (99.6–100.0%), respectively. Misclassification was most common for deaths with primary ICD codes for endocrine‐metabolic and genito‐urinary diseases.
National mortality coding under‐estimated the true proportion of CHD and stroke deaths in the cohort by 13.6% and 50.8%, respectively.
Misclassification of cause of death may have implications for conclusions drawn from epidemiological research.
Administrative database research infrequently used validated diagnostic or procedural codes
2011, Journal of Clinical Epidemiology
Citation Excerpt :
When outcome variables are subject to nondifferential misclassification, association measures are typically biased toward the null [23]. When nondifferential effects variables used to define cohorts (by excluding prevalent disease at baseline), Pekkanen et al. [24]. demonstrated that association measures could be biased away from the null (with the degree of bias varying by disease incidence and prevalence).
Administrative database research (ADR) frequently uses codes to identify diagnoses or procedures. The association of these codes with the condition it represents must be measured to gauge misclassification in the study. Measure the proportion of ADR studies using diagnostic or procedural codes that measured or referenced code accuracy.
Random sample of 150 MEDLINE-cited ADR studies stratified by year of publication. The proportion of ADR studies using codes to define patient cohorts, exposures, or outcomes that measured or referenced code accuracy and Bayesian estimates for probability of disease given code operating characteristics were measured.
One hundred fifteen ADR studies (76.7% [95% confidence interval (CI), 69.3–82.8]) used codes. Of these studies, only 14 (12.1% [7.3–19.5]) measured or referenced the association of the code with the entity it supposedly represented. This proportion did not vary by year of publication but was significantly higher in journals with greater impact factors. Of five studies reporting code sensitivity and specificity, the estimated probability of code-related condition in code-positive patients was less than 50% in two.
In ADR, diagnostic and procedural codes are commonly used but infrequently validated. People with a code frequently do not have the condition it represents.
Exposure to substances in the workplace and new-onset asthma: an international prospective population-based study (ECRHS-II)
2007, The Lancet
Citation Excerpt :
We excluded people who reported a history of asthma or respiratory symptoms (wheezing or whistling without a cold; ever having had asthma; having been woken by an attack of shortness of breath in the past 12 months) in ECRHS. These exclusions were intended to avoid confounding by baseline health status that could be associated with future risk of disease and risk factors.19 Asthma was defined by the participant either having an asthma attack or use of asthma medication in the 12 months before interview.
The role of exposure to substances in the workplace in new-onset asthma is not well characterised in population-based studies. We therefore aimed to estimate the relative and attributable risks of new-onset asthma in relation to occupations, work-related exposures, and inhalation accidents.
We studied prospectively 6837 participants from 13 countries who previously took part in the European Community Respiratory Health Survey (1990–95) and did not report respiratory symptoms or a history of asthma at the time of the first study. Asthma was assessed by methacholine challenge test and by questionnaire data on asthma symptoms. Exposures were defined by high-risk occupations, an asthma-specific job exposure matrix with additional expert judgment, and through self-report of acute inhalation events. Relative risks for new onset asthma were calculated with log-binomial models adjusted for age, sex, smoking, and study centre.
A significant excess asthma risk was seen after exposure to substances known to cause occupational asthma (Relative risk=1·6, 95% CI 1·1–2·3, p=0·017). Risks were highest for asthma defined by bronchial hyper-reactivity in addition to symptoms (2·4, 1·3–4·6, p=0·008). Of common occupations, a significant excess risk of asthma was seen for nursing (2·2, 1·3–4·0, p=0·007). Asthma risk was also increased in participants who reported an acute symptomatic inhalation event such as fire, mixing cleaning products, or chemical spills (RR=3·3, 95% CI 1·0–11·1, p=0·051). The population-attributable risk for adult asthma due to occupational exposures ranged from 10% to 25%, equivalent to an incidence of new-onset occupational asthma of 250–300 cases per million people per year.
Occupational exposures account for a substantial proportion of adult asthma incidence. The increased risk of asthma after inhalation accidents suggests that workers who have such accidents should be monitored closely.

View all citing articles on Scopus

View full text

Original ArticleNondifferential disease misclassification may bias incidence risk ratios away from the null

Abstract

Background and Objective

Study Design and Setting

Results

Conclusion

Introduction

Section snippets

Effect of misclassification of disease on estimated incidence

Discussion

Acknowledgments

Am Heart J

Lancet

Bias due to misclassification in the estimation of relative risk

Am J Epidemiol

Exception to the rule about nondifferential misclassification

Am J Epidemiol

Transitions in an imperfectly observed binary variable: depressive symptomatology in adolescents

Stat Med

Methods for assessing difference between groups in change when initial measurements is subject to intra-individual variation

Stat Med

The effects of measurement error in response variables and tests of association of explanatory variables in change models

Stat Med

Childhood versus adulthood onset asthma

Am J Respir Crit Care Med

European Community Respiratory Health Survey II Steering Committee. The European Community Respiratory Health Survey II

Eur Respir J

Validation of questionnaire and bronchial hyperresponsiveness against respiratory physician assessment in the diagnosis of asthma

Int J Epidemiol

Triggering of sudden death from cardiac causes by vigorous exertion

N Engl J Med

Original Article
Nondifferential disease misclassification may bias incidence risk ratios away from the null