Article Text
PDF
Abstract
Objectives Retrospective exposure assessment remains a problematic aspect of population-based case–control studies. Different methods have been developed, including case-by-case expert assessment and job–exposure matrices (JEM). The present analyses compare exposure prevalence and risk estimates derived by different exposure assessment methods.
Methods In the context of a case–control study conducted in seven European countries, exposure was estimated for asbestos, diesel motor emissions (DME) and crystalline silica, using three different assessment methods. First, experts assigned exposures to all reported jobs on a case-by-case basis. Second, a population-specific JEM (PSJEM) was developed using the expert assessments of controls only, and re-applied to all study subjects. Third, an independent general population JEM (GPJEM) was created by occupational exposure experts not involved in the original study, and applied to study subjects. Results from these methods were compared.
Results There was poor to fair agreement in assigned exposure between expert assessment and the GPJEM (kappas: asbestos 0.17; DME 0.48; silica 0.38). Exposure prevalence was significantly heterogeneous (p<0.01) between countries for all three agents and assessment methods. For asbestos and DME, significant country heterogeneity in risk estimates was observed when using expert assessment. When applying the GPJEM, the heterogeneity in risk estimates for asbestos and, to some extent, silica diminished.
Conclusions It has been previously advocated that the expert assessment approach to assign exposures based on detailed questionnaire responses provides more accurate exposure estimates than JEM-based results. However, current results demonstrated little, if any, advantage of case-by-case assessment when compared to a JEM approach.
- Exposure misclassification
- multi-centre study
- asbestos
- epidemiology
- cancer
- exposure assessment
Statistics from Altmetric.com
What this paper adds
Retrospective assessment of occupational exposures in the general population is difficult due to the wide variety of workplaces.
It is common practice in population-based studies to assign occupational exposures through expert assessment, but the performance of expert assessment relative to other approaches has never been evaluated in large multi-centre case–control studies.
Our study indicates that the theoretical advantages of case-by-case expert assessment in multi-centre studies might be absent or minimal compared to a general population job–exposure matrix.
Introduction
Assessment of occupational exposures in the general population is difficult given the wide variety of jobs and workplaces in which the study population will have been employed.1 Several methodologies have been used for assessing (retrospectively) occupational exposures, such as case-by-case expert judgement, job–exposure matrices (JEMs), self-reported exposure or combinations of these methods.2
Assessment by experts (such as occupational hygienists or occupational physicians) is generally considered the most credible method for assessing occupational exposures in population-based case–control studies. It is thought to induce less misclassification as experts can take into account local differences in material usage, production processes and control measures.2–4 Moreover, exposure differences between individuals with similar jobs can be considered. In single-centre studies where only one expert or closely associated experts perform the exposure assessment, uniform assignment of exposures can be ensured. In multi-centre studies, however, there may be large differences in agreement between centres and/or countries as was shown by 't Mannetje et al.5 Successful assessment therefore depends on more than just the ability of each expert. When more experts are involved, as in multi-centre studies, successful assessment also depends on standardising the work of experts who operate independently in different settings.5
Although expert assessment might be the preferred method, it is nonetheless dependent on questionnaire responses which are susceptible to non-differential and differential reporting errors. Validity depends partly on the validity of the subjects' reporting and partly on the experts' judgements. It has been postulated that job-specific questionnaires do not provide the expected gain in accuracy as the subject's recall for job titles probably exceeds their ability to recall detailed information on work conditions decades ago.6 A JEM is a more standardised method to translate jobs into specific exposures.1 Scores in these matrices are often dichotomous (exposed vs non-exposed) or have a semi-quantitative character (low-medium-high exposure). A population-specific JEM (PSJEM) is based on exposure information obtained from the study population itself.7–9 By using only information from control subjects, the PSJEM can overcome potential recall and reporting bias. This might play a role in case-by-case expert assessment because cases might be more prone to report exposure than controls in a similar work environment, resulting in differential bias. A general population JEM (GPJEM) is independent of any study population and avoids both potential recall and reporting bias. However, a GPJEM is created by experts based on their understanding and experience of exposures; it ignores local conditions that may differ from those upon which the JEM was based. In contrast to individual assessment, matrices (PSJEM and GPJEM) allocate the same exposure estimate to all workers within a job category and do not take inter-individual variability into account. This represents a major drawback since it is known that there may be differences in exposure levels between individuals with the same job.10
The INCO Copernicus study was a large multi-centre case–control study that employed expert assessment to investigate the association between occupational exposure and lung cancer risk. Results have been reported previously,5 11–16 including a study on exposure misclassification by expert assessment.5 The evaluation of experts' assessment reliability through an inter-team agreement study demonstrated highly varying levels of agreement for different exposures of interest. For instance, Cohen's kappas ranged from −0.04 for polycyclic aromatic hydrocarbons to 0.93 for arc welding fumes.5 The INCO study offered a unique opportunity to compare different exposure assessment methods. Current analyses aimed to describe differences in exposure assignment and corresponding risk estimates for occupational lung carcinogens between countries arising from different methods in this multi-centre study. Expert assessment was therefore compared to a PSJEM and a GPJEM.
Methods
Data were obtained from a lung cancer case–control study (INCO Copernicus) which was conducted in six Central and Eastern European (CEE) countries (Czech Republic, Hungary, Poland, Romania, Russia and Slovakia) and the United Kingdom. Newly diagnosed cases of lung cancer were recruited during 1998–2002 together with controls, frequency matched on sex and age (±3 years). Previous publications describe subject selection in more detail.5 11–16
A face-to-face interview was conducted by trained interviewers to assess lifestyle factors and occupations held for more than 1 year (6 months for the UK population), hereafter called the ‘job period’. For each job period questions were asked on tasks performed, machines used and work environment. Additionally, a more detailed questionnaire was completed for the following 18 specified jobs or industries: motor vehicle mechanic, wood worker, painter, welder, toolmaker or machine tool operator, miner or quarryman, insulation worker, meat worker, farmer, and the chemical, coke manufacture, foundry, glass, printing, rubber, steel, tanning and asbestos compounds industries. In the UK the questionnaire was extended to include five extra jobs: dock worker, construction worker, driver, fire service, and cook and kitchen help.
Local industrial hygienists blinded to case–control status evaluated and coded the occupational information. The protocol was based on that used previously in Canada.17 Occupation was coded using the International Standard Classification of Occupations (ISCO), 1968.18 Exposure to 70 agents was assessed for each job period. Indices used were intensity, frequency and confidence of exposure. Intensity represented the exposure level (1=low, 2=medium and 3=high). Standardisation of the application of the assessment methodology was achieved through yearly workshops and coding exercises.5 Crystalline silica exposure from sand, concrete dust, cement dust and brick dust was calculated using an algorithm, based on expert evaluation of exposure to the source material.12 The algorithm was not country or time specific. In current analyses a job period was considered exposed when intensity was assessed as 1 or higher.
Job–exposure matrices
Two JEMs were created with ISCO 1968 (5-digit) codes on the job axis: a PSJEM and a GPJEM. For this exercise three known or suspected human lung carcinogens with relatively high exposure prevalence were selected: asbestos, diesel motor emissions (DME) and crystalline silica. Since current analysis aimed to compare exposed versus non-exposed jobs/subjects, time trends in exposure levels were not taken into account for the JEMs.
Population-specific JEM
A country-specific PSJEM was created based on the results of expert assessment within the INCO study. Only assessments for control subjects were used to control for recall and reporting bias. Exposure was assigned to an ISCO code when experts assessed exposure of intensity 1 or more to over 33% of the job periods within this ISCO code, by country. Therefore, a particular job (ISCO code) was considered non-exposed when exposure was reported for 33% or less of all job periods with this ISCO code. A job/country-specific assessment could not always be made due to the low prevalence of some job titles. In the case of missing country-specific assignments (ie, when there were no controls with that job title in a particular country; 9% of job periods) estimates were based on all countries combined. Some jobs were performed only by cases (2% of job periods); in these situations assignments were based on the exposure information of the cases.
General population JEM
A GPJEM (the ‘DOM-JEM’) was created with exposure assessments for every single 5-digit ISCO code. Three occupational exposure experts (HK, RV and SP) who were not involved in the original INCO study carried out the ratings, independently of any information from the study population. Exposure ratings were performed separately, with conflicting scores settled by consensus. The assigned intensity score was ordinal (0=no exposure, 1=low exposure and 2=high exposure). Initial agreement for the three raters was 77% (asbestos), 92% (DME) and 95% (silica). The DOM-JEM was developed for application in general population studies, and given that specific occupational exposures are relatively rare, exposure assessment was designed to be more specific than sensitive.1 In the current analyses, exposed jobs (intensity scores 1 or 2) were compared with non-exposed.
Statistical analyses and evaluation methods
Cohen's kappa statistic was calculated as a measure of inter-method agreement, giving a single measure of agreement beyond that expected by chance alone. Assignments using expert assessment, the PSJEM and the DOM-JEM were compared to each other, based on exposed versus non-exposed jobs. The kappa was interpreted using the following arbitrary cut points: <0.4 poor, 0.4–0.75 fair to good and >0.75 excellent agreement. Heterogeneity in exposure prevalence by country was tested with a χ2 test.
ORs for lung cancer and ever exposure to each of the three agents were calculated, adjusted for centre, age, smoking status, gender, and occupations and industries known to present excess risk of lung cancer (list A based on ISCO and industry coding19). Misclassification in the expert assessment is likely to be non-differential with respect to disease status, while for the two JEMs it is evidently non-differential. Since true exposures remained unknown, calculation of the sensitivity and specificity of the three methods was not feasible. However, when an association between exposure and disease exists, non-differential misclassification will generally cause underestimation of the association and a decrease in the power of statistical tests, that is it leads to an attenuation of the OR towards the null value of 1.20–22 Non-differential misclassification may produce both spurious heterogeneity or homogeneity across strata of a third variable (in this case country), due to differences in the stratum-specific frequency of misclassification.20 However, biological effects of agents are not likely to vary widely across countries. Similar risk estimates are expected for different countries assuming similar exposure intensity and exposure duration. We tested for heterogeneity in the association between exposure and disease by country via an interaction term between country and exposure (Wald based). Strength of associations and country heterogeneity of risk estimates were used to describe and compare the different methods. All analyses were performed using SAS V.9.1.
Results
Table 1 describes the subject and job characteristics of the study population. Current analyses concerned 4784 subjects, which is 80% of the total recruited between 1998 and 2002 (n=5979). Excluded were subjects without expert assessment due to insufficient job information (n=1009) or without existing ISCO codes (n=4), and controls with a diagnosis which could eventually be associated with smoking (n=182). The mean age of the population was 60 years with 82% being male. Distributions of age and gender were similar among cases and controls. Overall, 15 233 job periods were included for analyses.
Description of subjects and jobs in the INCO study
Percentages of job periods exposed to asbestos, DME and silica based on the different exposure assessment methods are presented in table 2. The exposure prevalence of asbestos and DME based on expert assessment and the PSJEM was generally lower compared to the prevalence obtained after applying the DOM-JEM. Particularly for asbestos there was about a fivefold difference in the CEE countries. This difference was much smaller in the UK. Exposure prevalence to silica was somewhat higher based on the expert ratings compared to the prevalence obtained using the DOM-JEM. Application of the PSJEM led to a slightly higher prevalence in most situations than with the original expert assessment. Irrespective of the method applied, significant heterogeneity existed between countries in exposure prevalence for each of the agents. Overall there was poor agreement between ratings by experts and the DOM-JEM for asbestos and silica, with kappas of 0.17 and 0.38, respectively. Kappas for asbestos varied from 0.04 to 0.25 between countries, and for silica from 0.15 to 0.54. No pattern could be distinguished regarding time period or country. Most discordance was present for miners and armed forces; only the DOM-JEM assigned asbestos exposure for several job types in these groups. Overall agreement for DME was fair to good with κ=0.48. Agreement in exposure assignments between the JEMs was comparable (asbestos κ=0.23, DME κ=0.64 and silica κ=0.35). Assignments from the expert assessment and the PSJEM showed overall excellent agreement for DME (κ=0.81), and fair to good agreement for asbestos (κ=0.53) and silica (κ=0.70).
Percentages of job periods exposed and agreement of exposure between three methods, stratified by country
Overall risk estimates for asbestos, DME and silica exposure in relation to lung cancer did not change notably when the different methods were applied: asbestos 1.04–1.19, DME 1.05–1.08 and silica 1.24–1.26 (table 3). Formal statistical significance (p<0.05) was reached for all overall risk estimates for silica. For asbestos statistical significance was reached only for the analyses based on exposure estimates derived from the DOM-JEM. Furthermore, asbestos results from the DOM-JEM appeared to be less heterogeneous (p=0.44) between countries compared to the other methods (with p values for heterogeneity below 0.05). Risk estimates for DME exposure were comparable for all methods applied, each with significant heterogeneity in country-specific risk estimates (p<0.01). Risk estimates for silica exposure were quite homogeneous across countries, except when applying the PSJEM. Epidemiological results from the original expert assessment and the DOM-JEM were similar for silica, although the latter method appeared to result in slightly less heterogeneity in country-specific ORs. Restricting analyses to highly exposed jobs only (experts' intensity score=3, DOM-JEM score=2) showed similar results.
Risk estimates for lung cancer: outcomes of expert assessment and job–exposure matrices
Discussion
The present findings pertain to a multi-centre study and may not necessarily be applicable to single-centre studies. We think, however, that these results provide valuable insight in the performance of different exposure assessment methodologies which should be considered when starting new studies.
The INCO study showed large heterogeneity in exposure prevalence between countries. This was observed when applying both the expert assessment and JEMs. Since these were relatively small samples of the general population from areas which are likely to vary in economic activity, this is not surprising. However, some of this heterogeneity could be due to exposure misclassification. Previous evaluations of the reliability of expert assessment within INCO demonstrated that the quality of the individual expert teams was comparable, but that misclassification levels differed greatly between the exposures of interest assessed. Poor agreement was reported for asbestos and silica, with Cohen's kappas of 0.27 and 0.15, respectively. For DME the agreement was excellent (κ=0.8).5 Another reliability study of expert ratings reported similar kappas around 0.70.23 However, in this study raters had been working together for a decade, which is not feasible in multi-centre studies. Note, however, excellent agreement does not automatically imply that assigned exposure levels are necessarily more accurate.
Inter-rater concordance provides little information on the validity of estimates of exposure prevalence. Nonetheless, low agreement between experts for some exposures of interest suggests that there is potential for considerable misclassification in exposure assignment by experts. In addition, reporting bias could be an issue when job information is obtained from subjects, since cases and controls might report exposures and circumstances differently. Applying a JEM, which links job titles with exposures in a systematic and unbiased way,22 can minimise this type of bias. However, the performance of JEMs is generally limited by inability to account for variability in exposures within jobs (from factory to factory and worker to worker).24 25 Nevertheless, agreement between expert assessment and the DOM-JEM was in the same range as the inter-rater agreement found in previous analyses for asbestos and silica.5 Only for DME was it somewhat lower (κ=0.48 compared to 0.80 between experts). This corroborates the view that the accuracy of exposure assessment depends, to some extent, on the type of exposure of interest. The agreements between expert assessment and the PSJEM showed that the expert ratings for DME (κ=0.81) and silica (κ=0.70) were closely related to job title, while asbestos (κ=0.53) showed more variation. This implies that exposure assessment for asbestos was based more on specific circumstances and material used than on job title alone.
We observed substantial heterogeneity in country-specific risk estimates based on expert assessment for asbestos and DME. This might be caused by regional quantitative and qualitative differences in exposure circumstances. In the case of asbestos, exposure circumstances could vary because of different fibre types, possibly with different carcinogenicity potential11 26 27 as well as different average concentrations and durations of exposure, and co-exposures. When the UK was excluded from the analyses, heterogeneity in risk estimates for asbestos decreased for the expert assessment (p=0.27), indicating that the UK caused most of the heterogeneity in this study for asbestos. The UK used additional questionnaires, but excluding the extra jobs covered by the questionnaires from the analysis did not affect the observed heterogeneity in risk estimates. Notwithstanding the methodological differences, the biological effect of carcinogens per se is not expected to vary widely across countries. As risk estimates did differ in the INCO study, one could argue that this is at least partly caused by inter-rater differences especially as the country heterogeneity in risk estimates appeared to diminish when using the DOM-JEM. More homogeneous risk estimates for silica were derived for the expert assessments compared to the other two methods; this might be the result of standardisation by the algorithm applied in the INCO study.12
Considerable efforts were made in the INCO study to optimise exposure assessment by collecting data in a standardised manner across countries, face-to-face interviews with structured questionnaires, and intensive training of local experts. Nevertheless, expert assessment does not appear to result in stronger associations for the three known lung carcinogens than the DOM-JEM. Both methods gave similar results as to whether occupational exposure to asbestos, DME and/or silica increases lung cancer risk. Yet the DOM-JEM was faster and cheaper. Furthermore, interpretation of the epidemiological results might well have been different because of less heterogeneous risk estimates across countries. The debate concerning country differences regarding asbestos as described by Carel and co-workers11 would not necessarily have been brought up as the argument was mostly based on the observed heterogeneity in risk estimates and absence of elevated lung cancer risk in the CEE countries associated with the predominant use of chrysotile in those countries. Recently a Canadian case–control study based on expert assessment showed an increased risk of lung cancer following asbestos exposure (OR 1.78, 95% CI 0.94 to 3.36),28 while over 90% of asbestos used in North America is chrysotile. Based on the presented analyses one could argue that the absence of a risk in CEE countries might well have been caused by exposure misclassification and should not necessarily have been interpreted mechanistically.
It is important to note that the present results pertain to a multi-centre study and it remains unknown to what extent these findings can be generalised to other studies, other experts and other GPJEMs. Method validity can depend, inter alia, on: (a) the skill and experience of the experts or GPJEM designers; (b) the time taken to train experts and the time invested in the entire process of expert assessment or JEM design; and (c) the quality and number of documentary sources available. Nevertheless, the present study is one of the few that have compared the epidemiological results of different exposure assessment methodologies in a case–control study and provides some insights into their performance and needs to be extended in other studies so as to assess the generalisability of our observations.
In conclusion, results suggested heterogeneity between countries in the prevalence of exposure to occupational carcinogens across Europe. For asbestos and, to some extent, for silica exposure the observed heterogeneity in risk estimates largely disappeared when using the DOM-JEM. The hypothesis that identifying relatively small risks in a general population study is most likely to be achieved by expert assessment based on extensive job information is appealing. However, the theoretical advantages of case-by-case expert assessment in a multi-centre study might be minimal since similar overall results with less heterogeneity could have been achieved faster and cheaper by applying an external GPJEM.
Acknowledgments
We are thankful to Dr Jack Siemiatycki for his helpful comments and input.
References
Footnotes
This manuscript was written on behalf of the INCO group (the principal investigators of the original studies): David Zaridze, Neonila Szeszenia-Dabrowska, Peter Rudnai, Jolanta Lissowska, Eleonora Fabianova, Dana Mates, Vladimir Bencko, Lenka Foretova, Vladimir Janout, Joelle Fevotte, Tony Fletcher and Paul Brennan.
Current analyses were performed within the scope of the SYNERGY project. SYNERGY is a pooled analysis of case–control studies on the joint effects of occupational carcinogens in the development of lung cancer, which started in 2007. This project is coordinated by the International Agency for Research on Cancer (IARC).
Funding This study is funded by the German Social Accident Insurance (DGUV).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.