Article Text

Download PDFPDF

Comparison of expert and job-exposure matrix-based retrospective exposure assessment of occupational carcinogens in the Netherlands Cohort Study
  1. Nadine S M Offermans1,
  2. Roel Vermeulen2,3,
  3. Alex Burdorf4,
  4. Susan Peters2,
  5. R Alexandra Goldbohm5,
  6. Tom Koeman2,
  7. Martie van Tongeren6,
  8. T Kauppinen7,
  9. Ijmert Kant8,
  10. Hans Kromhout2,
  11. Piet A van den Brandt1
  1. 1Department of Epidemiology, GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre, Maastricht, The Netherlands
  2. 2Division of Environmental Epidemiology, Institute for Risk Assessment Sciences, Utrecht University, Utrecht, The Netherlands
  3. 3Julius Center for Health Sciences and Primary Care, University Medical Center, Utrecht, The Netherlands
  4. 4Department of Public Health, Erasmus MC, Rotterdam, The Netherlands
  5. 5Department of Prevention and Health, TNO Quality of Life, Leiden, The Netherlands
  6. 6Institute of Occupational Medicine, Edinburgh, UK
  7. 7Department of Occupational Health, Finnish Institute of Occupational Health, Helsinki, Finland
  8. 8CAPHRI School for Public Health and Primary Care, Department of Epidemiology, Faculty of Health, Medicine and Life Sciences, Maastricht University Medical Center, Maastricht, The Netherlands
  1. Correspondence to Nadine S M Offermans, Department of Epidemiology, Maastricht University Medical Centre, P.O. Box 616, 6200 MD Maastricht, The Netherlands; nadine.offermans{at}


Objectives Reliable retrospective exposure assessment continues to be a challenge in most population-based studies. Several methodologies exist for estimating exposures retrospectively, of which case-by-case expert assessment and job-exposure matrices (JEMs) are commonly used. This study evaluated the reliability of exposure estimates for selected carcinogens obtained through three JEMs by comparing the estimates with case-by-case expert assessment within the Netherlands Cohort Study (NLCS).

Methods The NLCS includes 58 279 men aged 55–69 years at enrolment in 1986. For a subcohort of these men (n=1630), expert assessment is available for exposure to asbestos, polycyclic aromatic hydrocarbons (PAHs) and welding fumes. Reliability of the different JEMs (DOMJEM (asbestos, PAHs), FINJEM (asbestos, PAHs and welding fumes) and Asbestos JEM (asbestos) was determined by assessing the agreement between these JEMs and the expert assessment.

Results Expert assessment revealed the lowest prevalence of exposure for all three exposures (asbestos 9.3%; PAHs 5.3%; welding fumes 11.7%). The DOMJEM showed the highest level of agreement with the expert assessment for asbestos and PAHs (κs=0.29 and 0.42, respectively), closely followed by the FINJEM. For welding fumes, concordance between the expert assessment and FINJEM was high (κ=0.70). The Asbestos JEM showed poor agreement with the expert asbestos assessment (κ=0.10).

Conclusions This study shows case-by-case expert assessment to result in the lowest prevalence of occupational exposure in the NLCS. Furthermore, the DOMJEM and FINJEM proved to be rather similar in agreement when compared with the expert assessment. The Asbestos JEM appeared to be less appropriate for use in the NLCS.

  • Population-based study
  • occupational exposure
  • expert
  • job-exposure matrix
  • agreement
  • retrospective exposure assessment
  • cancer
  • hygiene/occupational hygiene
  • epidemiology
  • biological monitoring
  • longitudinal studies
  • genotoxicity
  • exposure assessment
  • dermal exposure
  • rubber
  • dioxins
  • diesel fumes
  • benzene
  • agriculture
  • leukaemia
  • haematology
  • contact dermatitis
  • back disorders
  • ergonomics
  • male reproduction
  • electromagnetic fields
  • wood dust
  • bronchitis
  • health and safety

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

What this paper adds

  • Retrospective occupational exposure assessment remains a challenge in most population-based studies, as accurate exposure measurements are most of the time not at hand.

  • Reliability of a JEM depends on the study design, exposure of interest and quality of work history/exposure information available.

  • Prevalence estimates in the Netherlands Cohort Study (NLCS) were lower when assessed by case-by-case expert assessment as compared with JEMs.

  • This study indicates that the DOMJEM and FINJEM proved to be rather similar in agreement when compared with the expert assessment, while the Asbestos JEM appeared to be less appropriate for use in the NLCS.


Reliable and valid assessment of occupational exposure to carcinogens is critical in the conduct of occupational epidemiological studies, as misclassification of exposure might bias and attenuate the risk estimate.1 ,2 Several methodologies exist for estimating occupational exposures retrospectively from the available information on occupational histories,2 of which case-by-case expert assessment and job-exposure matrices (JEMs) are commonly used.3

Case-by-case expert assessment is generally considered the best possible method for assessing occupational exposure in population-based studies.2 ,4 Availability of detailed descriptions of industry, company, working environment, tasks and products used enables experts to allow for local differences in material usage, production processes, control measures and/or calendar period of exposure.4 This enables them to take account of exposure differences between individuals with similar jobs, which can result in less misclassification of exposure. Possible disadvantages of this approach are the learning phenomenon (as experts' skills evolve during the process)1 ,5 and inter-expert variability within studies and between studies (as their training, experience and work field might be different).4 ,6 In addition, case-by-case expert assessment requires considerable resources.5

A JEM is a cross-classification of job codes and/or industries on one axis and exposure agents on the other, with the cells of the matrix indicating the presence, intensity and/or prevalence of exposure to a specific agent in a specific job code/industry. In addition, some JEMs contain a third axis indicating time period.2 Unlike expert assessment, JEMs allocate the same exposure estimate to all workers within a job code, thereby disregarding possible inter-individual variability within job codes. This is a major drawback since there may be large differences in exposure levels between individuals with the same job in the same company and/or different companies. Conversely, the main advantage of using a JEM lies in the more standardised way in which jobs are being translated into specific exposures (ie, computerised linkage of exposure estimates to job codes), representing a more efficient and reproducible methodology.4 In addition, some JEMs, like the FINJEM, might also provide a more transparent methodology compared with case-by-case expert assessment as exposure measurements (though small in number) underlie the exposure estimate of (several of) the matrix cells'.7

In most population-based studies, actual exposure levels remain unknown due to the absence of exposure measurements,2 though recent efforts have made use of existing large amounts of exposure data.8 ,9 Moreover, study subjects are employed in a wide variety of jobs and workplaces, which renders (retrospective) assessment of occupational exposure extremely difficult for most, if not all population-based studies.10 ,11 One of these population-based studies is the prospective Netherlands Cohort Study (NLCS), which started in 1986 among 120 852 men and women.12 ,13 For a random sample of this cohort (the subcohort), case-by-case expert assessment, executed in the framework of a previous study in the NLCS,13 is available for exposure to asbestos, polycyclic aromatic hydrocarbons (PAHs) and welding fumes.

As expert assessment is not possible for the cohort as a whole, the objective of the present study was to evaluate several candidate JEMs in terms of reliability, using the case-by-case expert assessment of the subcohort as the ‘gold standard’ for the selection of the most appropriate JEM. Therefore, in this study, reliability refers to method agreement: ‘Do two techniques used to measure a particular variable, in otherwise identical circumstances, produce the same result?’.14 The JEMs compared were the Asbestos JEM15 and DOMJEM,4 developed in the Netherlands, and the Finnish FINJEM.7 The reason for considering a Finnish JEM was that the FINJEM has been used in a previous study in the NLCS.16


Study population

The study design and data collection strategies have been described in detail previously. In brief, the NLCS started in September 1986 when 58 279 men and 62 573 women aged 55–69 years, originating from 204 municipalities with computerised population registries, were enrolled in the cohort. At baseline, participants completed a self-administered questionnaire on dietary habits and lifestyle, occupational history and other potential risk factors for cancer.12

Following the case-cohort approach, a random subcohort was selected immediately after baseline to estimate person-years at risk accumulated in the entire cohort.12 As the number of long-term employed women was rather low, the current analyses were restricted to men.

For these male subcohort members (N=1630), case-by-case expert assessment was available in the framework of a previous study in the NLCS for exposure to asbestos, PAHs and welding fumes.13 Therefore, the current study was conducted on this subcohort only.

The NLCS was approved by the institutional review boards of the Netherlands Organisation for Applied Scientific Research TNO (Zeist) and Maastricht University (Maastricht).

Methods of assessment of occupational exposure

Information on lifetime occupational history until 1986 was obtained from the questionnaire filled in at study enrolment. Questions concerned the job title, name and type of the company, products made in the department and period of employment. Based on these questions, occupations were coded according to the Standard Occupational Classification of 1984 of the Dutch Central Bureau of Statistics (CBS-84), supplemented by a three-digit code assigned within the NLCS (CBS-NLCS) based on the job title. Subjects could enter a maximum of five occupations which was generally sufficient to cover the lifetime occupational history for the large majority of the cohort, as cohort subjects held on average 1.9 job codes during their working life. For all subjects, the job code was assessed for each of the maximally five occupations held between starting work and 1986.

Case-by-case expert assessment

For the male subcohort members in the NLCS, the combined case-by-case expert assessment of two independent expert raters (an occupational hygienist, IJ Kant, and an occupational epidemiologist, Gerard MH Swaen) is available, among others, for exposure to asbestos, PAHs and welding fumes.13 This expert assessment consisted of three steps. First, both experts excluded, independent of each other, all CBS-NLCS job codes with no potential exposure to these carcinogens. Second, for all job codes with potential exposure, the complete self-reported information on job title, name and type of the company, products made in the department and period of employment was used for scoring the prevalence (P) of exposure to these carcinogens in the Netherlands, as the prevalence may differ between companies and periods. This scoring was also carried out independent of each other. Four exposure prevalence categories were defined: no exposure to the specific carcinogen, possible exposure (prevalence of exposure estimated to be <30%), probable exposure (prevalence of exposure of 30%–90%) and nearly certain exposure (prevalence of exposure >90%). In a final expert meeting, all job codes for which at least one of the experts suspected relevant exposure were re-evaluated and a consensus prevalence of exposure was assigned to each of the job codes within the NLCS. In this assessment, the intensity (I) of exposure was not taken into account. It may be expected that those job codes with a high prevalence of exposure also entail a high intensity of exposure, but a secondary analysis of FINJEM for asbestos and PAHs showed that this is not an unequivocal pattern. For job codes with a low prevalence of exposure, intensity of exposure is often less clear. Table 1 shows an overview of relevant characteristics.

Table 1

Characteristics of the case-by-case expert assessment, Asbestos JEM, DOMJEM and FINJEM


The Dutch Asbestos JEM and DOMJEM and the Finnish FINJEM differ from each other regarding their axes. For an overview of relevant characteristics, see table 1.

The Asbestos JEM is a partly disease-oriented JEM as it is based on known asbestos exposure through available asbestos measurements (partly from the FINJEM Database) and verified cases of mesothelioma within occupations.15 On the occupational axis, the Asbestos JEM follows CBS-84 and includes only the 123 job codes with definite exposure to asbestos, out of the total of 914 job codes included in CBS-84. The exposure axis contains the agent asbestos, for which estimates of the prevalence (P) and intensity (I) of exposure are present in a categorical manner (four categories for P; seven categories for I). The time axis in the Asbestos JEM consists of ten 5 year time periods covering the period 1945–1994.

The DOMJEM is a generic JEM developed by experts in the Netherlands for application in general population studies. Given that specific occupational exposures are relatively rare, the slightest deviation of perfect specificity will lead to an (marked) underestimation of the degree of association. Therefore, the experts scored occupations, based on their knowledge and experience, with an emphasis on specificity rather than sensitivity.4 The DOMJEM uses ISCO-68 (International Standard Classification of Occupations of 1968) and contains 1842 different job codes. The relevant agents for this study included in the exposure axis are asbestos and PAHs, and the accompanying measure of exposure is a combined measure of P×I, for which three categories are available in the DOMJEM. As exposure is an estimation of the average exposure over time, this JEM does not possess a time axis. However, asbestos exposure levels assigned pertain to the period before the ban.

The FINJEM was constructed for exposure assessment in large register-based studies and is based on both expert assessment and exposure measurements.7 The FINJEM contains all 311 classes of the Combined Occupational Classification of Finnish Censuses in 1970–1985. The relevant agents covered by the exposure axis are asbestos, PAHs and welding fumes for which three continuous measures of exposure are available: P, I as well as a combined measure of P×I. This JEM includes a time axis consisting of four time periods, dissimilar in length, covering the period 1945–1994. FINJEM also includes five additional 3-year periods covering the years 1995–2009, but these were not used in the present study.

Comparison of exposure assessment methods

The main focus of the comparison was to study if the JEMs assign exposure to subjects similar to the reference method, case-by-case expert assessment and to each other. As the risk analyses within NLCS are primarily based on estimates of cumulative exposure (CE), CE per subject was chosen as the unit of comparison. Although several exposure measures are available (ie, prevalence and intensity), none of them are available for all methods: the DOMJEM only contains P×I and the case-by-case expert assessment only includes P. Given that case-by-case expert assessment only includes P, this measure was compared with P×I in the JEMs.

For the case-by-case expert assessment, the CE per subject is a combined measure of the prevalence (P) and duration (years) of exposure. To calculate the CE for the expert assessment, a weight was assigned to each category of exposure prevalence. Each weight corresponds to the midpoint of prevalence in each exposure category: no exposure, weight 0; possible exposure, weight 0.15; probable exposure, weight 0.60 and nearly certain exposure, weight 0.95. For each carcinogen, the CE per subject was calculated by multiplying the weight given to each exposure category by the number of years exposed in a certain job code and subsequently summing up all weighted exposures.13

For the JEMs, the CE is a combined measure of the prevalence, intensity (I) and duration (years) of exposure. To arrive at the CE for the JEMs, first the CBS-NLCS codes had to be recoded to the job classification used in the job axis of the JEMs (CBS-84 for the Asbestos JEM, ISCO-68 for the DOMJEM and Finnish codes for the FINJEM) in order to be able to link the JEMs. For that reason, the CBS-NLCS codes in the subcohort were merged into the broader classes of CBS-84. Furthermore, an algorithm recoding from CBS-NLCS to the five-digit ISCO-68 has been developed for the NLCS.17 Finally, an algorithm recoding ISCO-68 to the Finnish job codes had already been developed within the framework of the INTEROCC Study (Van Tongeren M, Kincl L, Richardson L, et al, paper presented at EPICOH 2011, Oxford). After recoding of the job codes, the actual linkage with the JEMs was carried out.

In order to obtain the CE for the Asbestos JEM and FINJEM, first P×I per job code was estimated using the time-specific exposure information in the time axis of both JEMs, before summing P×I over the maximum of five occupation subjects could enter. For those workers who started working before 1945, exposure estimates from the earliest time period in both JEMs were used, since the period before 1945 was absent in both JEMs.

For the DOMJEM, the CE was estimated by summing the product of P×I and duration for the maximum of five job codes. The DOMJEM scores of no, low and high exposure for P×I were arbitrarily assigned values of 0, 1 and 4 to mirror the log-normal (multiplicative) nature of occupational exposure levels. The weighting was based on reported levels for semi-quantitatively scored exposure, thereby assuring a balanced weighting between intensity and duration in the calculation of CE.18

Statistical analyses

In the subcohort, 1455 of the 1630 men completed the self-reported occupational history sufficiently to assign job codes. For these male subcohort members, the agreement in exposure assessment between the JEMs mutually and the case-by-case expert assessment was estimated. First, the prevalence of exposure among the male subcohort members according to the different methods was calculated as described in the previous section. The CE was categorised in four categories: non-exposed and three categories for the exposed based on the tertiles of CE. In order to determine the prevalence of exposure, the tertiles of CE were combined into one category of exposed (ie, CE>0). Second, weighted Cohen's κs and 95% CIs were determined for the four categories of CE using Stata SE V.10.0 (Stata). Linear weights were applied to take into account the extent of disagreement. These weights are proportional to the number of categories, with perfect agreement along the main diagonal (weight of 1) and smaller weights for the other cells (0.5 for ‘no vs low exposure’ and ‘low vs high exposure’), the smallest weight being for the biggest disagreement (0 for ‘no exposure vs high exposure’). κs were interpreted using the following arbitrary cut-off points: <0.4 poor, 0.4–0.75 moderate to good and ≥0.75 excellent agreement.

The following comparisons were made:

  1. Case-by-case expert assessment versus FINJEM (for exposure to asbestos, PAHs and welding fumes).

  2. Case-by-case expert assessment versus DOMJEM (for exposure to asbestos and PAHs).

  3. Case-by-case expert assessment versus Asbestos JEM (for exposure to asbestos).

  4. FINJEM versus DOMJEM (for exposure to asbestos and PAHs).

  5. FINJEM versus Asbestos JEM (for exposure to asbestos).

  6. DOMJEM versus Asbestos JEM (for exposure to asbestos).

Sensitivity analyses

Besides the main analysis where the unit of comparison concerned the individual CE for all male subcohort members, agreement between the different methods was also assessed restricted to exposed subjects in order to determine how the methods agree on the actual level of exposure. In order to determine if a subject was exposed, case-by-case expert assessment was used. Second, a comparison was made solely on P for FINJEM and the Asbestos JEM with the expert assessment. Third, individual job codes were used as the unit of comparison instead of the male subcohort members, with the CE per job code as the measure of interest for all methods. Fourth, an advantage of this approach included the possibility to study how agreement changes by time period, by assessing agreement on CE per job code for the periods 1929–1944, 1945–1959, 1960–1974 and 1975–1986 separately instead of focusing on the cumulative lifetime occupational exposure. As exposure prevalence in the expert assessment was estimated for the years in which the male subcohort members practiced their jobs, it was not possible to calculate a mean expert-based exposure for the abovementioned periods. Therefore, only the job codes of which the start and end dates concurred with the abovementioned time periods could be taken into the analyses.


Subject characteristics

The 1455 male NLCS subcohort members (mean age of 61.3 years) worked on average 16.6 years in the same job code (CBS-NLCS) and held on average 1.9 job codes during their working life, resulting in 1036 different CBS-NLCS codes from a total of 14 154 job codes. These CBS-NLCS codes were recoded into the respective ISCO-68 code (n=508), Finnish code (n=236) and CBS-84 code (n=78 of the 123 codes that entail asbestos exposure) to allow linkage to the DOMJEM, FINJEM and Asbestos JEM, respectively. The timeframe during which subjects were occupationally active, as reported in the questionnaire, includes the period 1929–1986. Furthermore, 40.8% of the male subcohort members started working before 1945.

Exposure prevalence

For exposure to asbestos, case-by-case expert assessment had the lowest prevalence of exposure (9.3%), followed by the FINJEM (28.3%), DOMJEM (30.0%) and Asbestos JEM (32.9%). For exposure to PAHs, case-by-case expert assessment also showed the lowest prevalence of exposure (5.3%), more closely followed by the DOMJEM (8.5%) and FINJEM (10.1%). The lowest prevalence of exposure to welding fumes was also attained by the case-by-case expert assessment (11.7%), with FINJEM resulting in a prevalence of 15.6%.

Agreement in exposure assessment: weighted Cohen's κ

Weighted Cohen's κs and 95% CIs for exposure assigned to the male subcohort members by the different methods are shown in table 2.

Table 2

Weighted Cohen's κs and 95% CIs for cumulative exposure* assigned to the male subcohort members (N=1455) by the different methods

For exposure to asbestos, the method showing the highest level of agreement with case-by-case expert assessment was the DOMJEM (Cohen's κ=0.29, 95% CI 0.23 to 0.32), followed by the FINJEM (Cohen's κ=0.23, 95% CI 0.19 to 0.29). The agreement between the expert assessment and the Asbestos JEM was low (Cohen's κ=0.10, 95% CI 0.05 to 0.13). Furthermore, the comparison between DOMJEM and FINJEM revealed the highest concordance between methods (Cohen's κ=0.50, 95% CI 0.47 to 0.55). Finally, the level of agreement with the Asbestos JEM was higher for FINJEM (Cohen's κ= 0.42, 95% CI 0.38 to 0.47) than for DOMJEM (Cohen's κ=0.25, 95% CI 0.20 to 0.29).

Patterns of agreement were comparable for exposure to PAHs. The DOMJEM showed the highest level of agreement with the case-by-case expert assessment (Cohen's κ=0.42, 95% CI 0.29 to 0.52), as was the concordance between DOMJEM and FINJEM highest (Cohen's κ=0.51, 95% CI 0.41 to 0.57).

For exposure to welding fumes, the level of agreement between case-by-case expert assessment and FINJEM was high (Cohen's κ=0.70, 95% CI 0.66 to 0.74).

Spearman's correlation analyses on the continuous CE measures revealed a similar relative ranking of the different methods with respect to each other, though absolute numbers were somewhat higher (results not shown).

Sensitivity analyses

The agreement between the different methods when restricting analyses to exposed subjects only was overall lower than in the main analyses, with the FINJEM and Asbestos JEM revealing hardly any agreement with the case-by-case expert assessment for asbestos (Cohen's κ=0.09, 95% CI −0.03 to 0.16 and Cohen's κ=0.04, 95% CI −0.06 to 0.12, respectively). For PAHs, FINJEM showed the highest agreement with the expert assessment (Cohen's κ=0.32, 95% CI 0.22 to 0.44) and equal agreement with the DOMJEM compared with the main analyses (Cohen's κ=0.51, 95% CI 0.36 to 0.64). For welding fumes, agreement between the FINJEM and expert assessment was lower compared with the main analyses (Cohen's κ=0.42, 95% CI 0.37 to 0.54) (see table 3).

Table 3

Sensitivity analysis: weighted Cohen's κs and 95% CIs for cumulative exposure* assigned by the different methods to exposed subjects only

When comparing the FINJEM and Asbestos JEM on P with the expert assessment, patterns of agreement were similar to the comparison on P×I in the main analyses (results not shown). Also when studying concordance between the methods when using individual job codes as the unit of comparison, instead of subjects as in the main analyses, differences were negligible (results not shown). Finally, results were again largely comparable to the main analyses if the agreement was determined for specific time periods instead of the cumulative lifetime occupational exposure (results not shown). Only for the last period 1975–1986, agreement between all methods was lower, though numbers were low because not all job codes could be included as the start and end dates of a job code did not necessarily concur with the time periods used.


Results of epidemiological studies should be interpreted in the light of the quality of the exposure assessment methods used and, if available, information on the validity of the assessment method should be included in the scientific report.19 ,20 As validation of retrospective exposure assessment is not easily accomplished because no gold standard is available, this study focused on reliability. Given the size of the study population, future exposure assessment in the NLCS will have to be based on JEMs. As reliability of a JEM may depend on the study design, exposure of interest and quality of work history/exposure information available, it is of paramount importance to apply different JEMs in the same cohort in order to evaluate consistency of findings in relation to generalisibility. Therefore, this study scored several JEMs on reliability in the NLCS by assessing agreement between JEMs and case-by-case expert assessment by means of Cohen's κ and the prevalence of exposure. To this end, case-by-case expert assessment was available for the subcohort in the NLCS.

First, results of this study show that the case-by-case expert assessment assigned the lowest prevalence of exposure. Second, the main analyses show variable agreement in exposure assignment, when comparing the JEMs with the expert assessment and with each other. For the comparison with the case-by-case expert assessment, the JEMs show poor agreement for asbestos (all JEMs), moderate agreement for PAHs (DOMJEM and FINJEM) and high agreement for welding fumes (FINJEM).

For the JEM–JEM comparisons, the agreement is moderate to good when comparing the DOMJEM and FINJEM for asbestos and PAHs, moderate when comparing the Asbestos JEM with the FINJEM and low when comparing the Asbestos JEM with the DOMJEM.

Third, sensitivity analyses showed comparable results to the main analyses, except for the analyses restricted to exposed subjects only for which agreement between methods tended to be (much) lower as expected.

Case-by-case expert assessment, when based on detailed questionnaires or interviews, is often considered the best possible method for retrospective exposure assessment in population-based cohort studies.2 ,21 Yet, studies comparing expert assessment to objective measurements in urine or the work environment show strongly varying results. The same is true for studies examining agreement between experts, which found κs from 0 to 1.0 with a median of about 0.6.2 As such, expert assessment should not generally be regarded as a gold standard. In the case-by-case expert assessment in this study, only limited information was available, which may have further hampered appropriate assessment of exposure patterns. However, Post et al 22 found in their study that exposure estimates hardly improved when actual measurements became available to experts and Teschke et al 2 found variable results in their review as not all studies showed an increase in the quality of the expert assessments with an increase in the amount of data available. It is of interest to note that the expert assessment in this study showed the lowest exposure prevalence. Given that occupational exposure prevalence to carcinogens is low in most population-based studies, this would seem preferable, though lower prevalence estimates do not guarantee that the risk of false-positive estimates is also lower. As a recent study of Bhatti et al 23 found the opposite, that is, exposure prevalence determined by expert assessment was higher than assessed by means of a JEM, this again points out the importance of study-specific comparison of different retrospective exposure assessment methods.

The rather low agreement between the expert assessment and the Asbestos JEM could be explained by the concept behind this latter JEM, as it was originally designed to aid subjects with asbestos-related diseases in pursuing compensation in legal trials. This JEM is based on 86 cases of asbestosis and 710 mesothelioma cases within occupations in the Netherlands.24 Since mesothelioma is almost exclusively due to asbestos exposure,25 this JEM only includes those 123 job codes with definite historical exposure to asbestos. Furthermore, administrative employees (ie, white-collar workers) in companies manufacturing asbestos products were also among the mesothelioma cases, an occupational group normally not expected to be exposed to asbestos. Consequently, this partly disease-oriented JEM might be more sensitive than the two general population JEMs, which are more aimed at specificity than sensitivity because of the low exposure levels in the general population.4 ,7 Results were in line with this expectation as exposure prevalence was highest for the Asbestos JEM, though differences with the other JEMs were small.

Lack of a time axis could also be responsible for the variable agreement,2 ,4 ,10 ,26 as exposure levels have generally decreased over time because of better understanding of occupational hazards and subsequent regulations.27 ,28 Nonetheless, the DOMJEM, which lacks a time axis, scored equally compared with the FINJEM on agreement with the expert assessment and better than the Asbestos JEM. Possibly, time trends in exposure levels were smaller compared with the difference between low and high intensity, which made inclusion of a time axis of less importance.

Finally, the level of agreement between the FINJEM and case-by-case expert assessment might be influenced by true differences in technology and use of materials between countries.21 ,29 Finland had a more stringent occupational policy than the Netherlands, had an extensive asbestos textile industry as opposed to the Netherlands and had an asbestos mine that might explain the low level of agreement for asbestos between the FINJEM and expert assessment.

This study had several limitations. First, no data were available on inter-expert agreement. Therefore, although the case-by-case expert assessment proved to be the most stringent method in exposure assignment, no conclusions could be drawn as to the actual reliability of the expert assessment itself. Second, the NLCS is a population-based study with accordingly low exposure prevalence. This has consequences for calculating Cohen's κ, yielding imprecise estimates when exposure prevalence is low.6

Though agreement in the sense of Cohen's κ is informative about the reliability of a method, agreement depends on how exposure is defined. As expected, when restricting agreement to exposed subjects only, Cohen's κs were lower with some comparisons revealing no agreement at all. As already known, agreement between methods is better on the dichotomy exposed versus non-exposed than on actual levels of exposure. Moreover, as the relation of assessment by either method (case-by-case expert assessment or JEM) to true exposure is unknown, it is not possible to quantify the amount of exposure misclassification for each method, though misclassification will by definition be non-differential in cohort studies. Therefore, when using either method for risk prediction, it should be kept in mind that risk estimates will be attenuated. In order to further assess the reliability of the JEMs, their predictive ability will be studied using the case-cohort approach in a future study on exposure to asbestos and cancer risk in the NLCS.

In conclusion, this study shows case-by-case expert assessment to assign the lowest prevalence of exposure in the NLCS. Furthermore, the DOMJEM and FINJEM proved to be rather similar in agreement when compared with this expert assessment. Finally, the Asbestos JEM appeared to be less appropriate for use in the NLCS.


The authors are indebted to the participants of this study and further wish to thank S van de Crommert, Dr L. Schouten, J Nelissen, C de Zwart, M Moll, W van Dijk, M Jansen and A Pisters for their assistance with data-entry and/or data management; L Preller regarding use of the FINJEM; A Volovics and A Kester for statistical advice and H. van Montfort, T. van Moergastel, L. van den Bosch and R Schmeitz for programming assistance.



  • Funding This study was supported by a grant from ZonMw (grant 50-50-500-98-6153). The sponsor had no role in the study design; collection, analysis, and interpretation of data; writing process or decision where to submit the paper for publication.

  • Competing interests None.

  • Patient consent Obtained.

  • Ethics approval The NLCS has been approved by the institutional review boards of the Netherlands Organisation for Applied Scientific Research TNO (Zeist) and Maastricht University (Maastricht).

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement This study contains unpublished data for the Spearman's correlation coefficients of the continuous cumulative exposure estimates. These data can be obtained from the corresponding author of this article.