Table 4

Validity and reliability of exposures estimated by experts

Authors, yearStudy populationExpert measureComparison measureMethod of assessing validity or reliabilityResults
Woitowitz et al, 197070Employees of a raw asbestos processing plant in GermanyStudy personnel ranked employees' asbestos exposure on a 4-point scale, based on tours of the worksite, and consultations with department heads, trade union personnel, plant or government physician, firm safety officer, and shop committee61 dust measurements- Comparisons of mean dust concentrations of expert-assessed exposure categories- Mean dust concentration for category 1 = 0.6 mg/m3; for category 2 = 1.2 mg/m3; for category 3 = 1.6 mg/m3
Goldberg et al, 198678Subjects of a case-control study among Canadian men with cancer at any of 19 tumor sitesA rater (occupational hygienist, chemist, or engineer) assessed subjects' presence or absence of exposures to 172 to 275 substances, based on information provided by subjects in a detailed interview, their own knowledge, review of bibliographic materials, and consultations with other expertsAssessments of 1 to 2 other raters on the study team using the same methods; and assessments by an expert from certain industries external to the study- % agreement between raters; kappa- Average % agreement in 6 inter-rater agreement trials ranged from 95.5% to 98.5%
- Average kappas ranged from 0.51 to 0.67
Kromhout et al, 198747Employees of a paint factory (n=29), a food processing facility (n=58), a nonwoven materials factory (n=164), and 2 coach works (n=144) in the NetherlandsTwo occupational hygienists ranked employees' exposures to either dust or solvents using a 4-point scale58 solvent measurements in the paint factory; and 421 dust measurements in the other plants- Proportion of variance in exposure explained (adjusted R2) by the hygienists' rankings- R2 ranged from 0.08 to 0.27 for dusts, median = 0.25
- R2 were 0.37 and 0.58 for the solvents, for the two hygienists
Ciccone and Vineis, 19887988 soft tissue sarcoma cases and 157 population based controls from a rice growing region of ItalyAn agricultural chemist with rice growing expertise assessed subjects' exposure to phenoxy herbicides using a 3-point scale, using information collected from the subjects on their job history, locations of farms, types of crops, characteristics of pesticide handling, and their subjective estimate of pesticide exposureAssessments by a similarly expert agricultural chemist, using the same method- % agreement between chemists; kappa- 95.5% agreement
- weighted kappa = 0.76
Hawkins and Evans, 19898012 chemical process workers in the US24 occupational hygienists with expertise in chemical processing randomly selected from 2 professional organizations gave quantitative estimates of the employees' toluene exposures, first based on a qualitatitive description of the process and work environment, then after viewing limited historical measurement data134 toluene exposure measurements- Comparison of mean, median, and 90th percentiles of exposure estimates to measured exposures- Mean measured exposure = 4.6 ppm; first estimate = 14 ppm; second estimate = 4.4 ppm
- Median measured exposure = 0.37 ppm; first estimate = 8.9 ppm; second estimate = 3.0 ppm
- 90th %ile of measured exposure = 16 ppm; first estimate = 41 ppm; second estimate = 21 ppm
Teschke et al, 198951225 Canadian sawmill workersThree pairs of occupational hygienists estimated hours of exposure to chlorophenate fungicides per year, after walk through survey of sawmillConcentration of total chlorophenate in urine samples, measured in 150 workers in the summer, and 154 workers in the fall- Proportion of variance in urinary chlorophenate concentrations explained by hygienists' estimates of hours of exposure (R2)- Lumber industry hygienists: R2 = 0.08; with skin exposure also included in the model, R2 = 0.26; government hygienists: R2 = 0.24; other industry hygienists: R2 = 0.22
- Intraclass correlation coefficients for agreement between members of each pair of hygienists- Lumber industry hygienists: ICC = 0.68; government hygienists: ICC = 0.40; other industry hygienists: ICC = 0.50
Post et al, 199181Employees in nine jobs in a small polyester factory in the Netherlands9 occupational hygienists each classified employees' exposures to styrene and methylene chloride into three absolute categories related to the Threshold Limit Value, using information about the process and jobs, and a visit to the plant; estimation was done two additional times after receipt of a small amount of measurement data45 styrene and 28 methylene chloride exposure measurements- Spearman rank correlation coefficient comparing relative ranking of jobs by each hygienist to measured ranking- Spearman r ranged from 0.3 to 0.9 for methylene chloride, median = 0.65; and from −0.4 to 0.65 for styrene, median = 0.2
- % agreement between absolute classifications of jobs by each hygienist and measured classifications- % agreement ranged from 0.15 to 1.0 for methylene chloride, median = 50%; and from 0.1 to 1.0 for styrene, median = 55%
Dovan et al, 19938281 homes of cases and controls in a study of childhood cancer in the USA study technician trained in a “wire coding” procedure classified the current configuration of the electrical transmission and distribution lines near each home into one of four ordinal categories in 1985- Repeat of the wire coding in 1990- % agreement in wire codes- 90% agreement
Macaluso et al, 19938329 paint department employees of a car assembly plant in the USOccupational hygienist (university or consultant) classified exposures of 695 job-department-year combinations to 6 product use groups and 7 specific chemical groups, into six absolute concentration categories, based on historical measurement dataAssessments by 4 similar hygienists, using the same method- Intraclass correlation coefficient for all job-department-year exposure scores and for cumulative exposure scores- ICCs for all job-department-year exposure scores ranged from −0.05 to 0.64, median = 0.14; ICCs for cumulative exposures from 0 to 0.85, median = 0.16
- ICCs for all job-department-year exposure scores ranged from −0.07 to 0.33, median = 0.24; ICCs for cumulative exposures from 0 to 0.58, median = 0.33
Takahashi et al, 19948442 deceased Canadian men who had cancer at various sites, all subjects in a case-control studyA team of experts (occupational hygienists, chemists, or engineers) ranked subjects' exposures to asbestos into 3 ordinal categories based on information provided by subjects in a detailed interview, their own knowledge, review of bibliographic materials, and consultations with other expertsMeasured asbestos fibre concentrations in 42 lung tissue samples taken at autopsy- Comparisons of mean lung asbestos concentrations for expert assessed exposure categories- Mean fibre concentration of subjects rated as having no asbestos exposure history = 0.09 f/ug dry lung; with low or moderate estimated exposure concentration = 0.14 f/ug; with high estimated exposure concentration = 8.7 f/ug
- Proportion of variance in lung asbestos concentration explained (R2) by experts' ratings- R2 = 0.32, with age also in model, p<0.0006
Armstrong et al, 19968531 cases of lymphohaematopoetic cancer and 124 controls who were subjects of a nested case-control study of petroleum marketing and distribution workers in CanadaStudy experts estimated exposures to total hydrocarbons and benzene, based on an algorithm which included mean measured exposure levels, and modifying information about the workplace, tasks performed, ambient environment, and products15 measurements of total hydrocarbon exposure in the 6 job-years, and 51 measurements of benzene exposure in the 9 job-years; these data were withheld from the expert estimation process- % difference in mean algorithm estimated exposures to withheld measurement means- % difference for total hydrocarbons ranged from −49% to 220%, median = −35%
- % difference for benzene ranged from −14% to 130%, median = 4.5%
de Cock et al, 19968615 fruit growing farms in the Netherlands15 occupational hygienists, pesticide experts, and fruit growers ranked potential for pesticide exposure by dermal and inhalation routes, of 14 tasks and of 15 spraying activities; done in three phases: after viewing a video on factors affecting exposure, after viewing slides about pesticide application tasks, and after reading written information on tasks and pesticidesMeasurements of airborne and dermal captan concentrations during spraying- Spearman rank correlation coefficients comparing relative ranking of activities by each expert to measured ranking- Spearman r ranged from −0.1 to 0.45 for inhalation exposure, median = 0.3; and from 0.03 to 0.9 for dermal exposure, median = 0.65
- Intraclass correlation coefficients for agreement between experts- ICCs ranged from 0.61 to 0.81 for inhalation exposure, median = 0.72; and from 0.53 to 0.71 for dermal exposure, median = 0.63
Segnan et al, 19968782 vineyard workers, 171 metal plating workers, and 158 leather goods workers in ItalyUsing occupational histories, industry specific questionnaires, lists of products used, and where available, exposure measurement data (in separate stages)The same experts' ratings using the full set of data available to them- Sensitivity and specificity using the ratings based on the full datatset as the gold standard- Sensitivities ranged from 0.13 to 0.99 (median = 0.78) using only the occupational histories, 0.05 to 1.0 (median = 0.76) using the industry-specific questionnaires, 0.87 to 1.0 (median = 0.97) using the product lists
- 8 agronomists assessed exposure to 10 pesticides, 6 classes of chemicals, and to the broad groups “fungicides” and “insecticides”- Specificities ranged from 0.12 to 0.90 (median = 0.52) using only the occupational histories, 0.17 to 0.99 (median = 0.77) using the industry-specific questionnaires, 0.77 to 1.0 (median = 1.0) using the product lists
- 8 industrial hygienists assessed exposures to 20 chemicals and 5 classes of chemicals
- 4 industrial hygienists assessed exposures to 20 solvents and 9 classes of chemicals
- Intraclass correlation coefficients for the agreement between raters- ICCs ranged from below 0 to 1.0 for all methods, with a median of 0.11 using only the occupational histories, 0.21 using the industry-specific questionnaires, 0.65 using the product lists, and 0.51 using monitoring data
Benke et al, 199788Jobs reported in a case-control study of glioma in Australia3 industrial hygienists and 2 occupational physicians assessed the presence or absence of exposure to 21 chemicals in 199 jobs randomly selected from subjects' histories and 49 jobs with exposure measurementsIndustrial hygiene reports for the 49 jobs from a database of surveys in the study region, over the period from 1978–1989- Sensitivity and specificity for exposure in 49 jobs with exposure measurements- Sensitivities ranged from 0.48 to 0.79, median = 0.65; specificities ranged from 0.91 to 0.98, median = 0.94
- Kappas for pairwise inter-rater agreement for the 199 jobs of study subjects- Kappas for inter-rater agreement ranged from 0 to 0.64, median = 0.19
- Kappas for intra-rater agreement based on reassessment of 50 of the 199 jobs, at least 4 months later- Kappas for intra-rater agreement ranged from 0.46 to 0.73, median = 0.60
McGuire et al, 199789179 job histories from population based case-control study of amyotrophic lateral sclerosis in the USPanel of four industrial hygienists, by consensus, rated exposure to three groups of chemical agents based on job history informationBlinded repeat assessment by same panel- % agreement and kappas- % agreement 90% for metals, 82% for solvents, and 97% for agricultural chemicals
- Kappas were 0.77 for metals, 0.64 for solvents, and 0.75 for agricultural chemicals
Semiatycki et al, 19979050 subjects from a case-control study among Canadian men with cancer at any of 19 tumour sitesConsensus review by two or more experts (industrial hygienists or chemists) of subjects' histories of their jobs, work environments, raw materials, products, and self reported exposures to assess presence and probability of exposure to 294 chemical and physical agentsTwo experts reassessed exposure to 94 jobs by consensus, and reassessed exposure to 92 different jobs independently from one another- Kappas- In consensus reassessment, weighted kappa for 4 categories of probability of exposure = 0.80; kappas for presence of exposure ranged from 0.51 to 0.94 for the 18 highest-prevalence substances, median = 0.75
- In independent reassessments, weighted kappas for 4 categories of probability of exposure ranged from were 0.73 and 0.76 for the two experts
Rybicki et al, 199891Job histories of 60 and 64 of 608 subjects in a case-control study of neurologic disease in the USExpert review by two industrial hygienists of self reported exposures to copper, lead and iron, of 60 and 64 study subjectsPrior expert review by one of the same industrial hygienists of self reported exposures of all 608 study subjects- Kappas for intra-rater and inter-rater agreement- Kappas = 0.26 for copper, 0.56 for lead, and 0.57 for iron, intra-rater
- Kappas = 0.15 for copper, 0.29 for lead, and 0.49 for iron, inter-rater
Cherrie and Schneider, 19999217 jobs in brick manufacturing in the UK, 13 jobs in rubber and pigment coating in the UK, 14 jobs in fibre reinforced plastics in Denmark, 13 jobs in an asbestos contaminated warehouse in the US, and 6 jobs in man made mineral fibre manufacturing in the USTwo industrial hygienists' estimates of exposure concentrations of respirable dust, toluene, styrene, asbestos, or man made mineral fibre, based on descriptions of the jobs, tasks, work environments, and control measures, using a structured assessment method based on emissions, processing at the source, and controlsMeasurements of airborne concentrations of respirable dust, toluene, styrene, asbestos, and man made mineral fibre- Correlation coefficient comparing log transformed exposure measurements to the hygienists' estimates- Pearson r ranged from 0.31 to 0.93 for all agents except styrene, and from 0 to 0.31 for styrene, for which there was little variability in measured exposure, median = 0.39
- Bias, as measured by ratio of the geometric mean exposure estimate to the geometric mean measured concentration- Bias ratio ranged from 0.47 (underestimate of exposure) to 2.86 (overestimate), median = 1.47
Tielemans et al, 199943Subjects of 2 case-control studies of male infertility in the NetherlandsResearchers used subject reported data from generic job histories and job specific questionnaires (additional data on tasks and substances used) to assess exposures to solvents and chromium- Urine samples analysed for metabolites of toluene and xylene (n=267) and for chromium (n=156)- Sensitivity and specificity in comparison to urine samples- Sensitivities = 0.30 using the generic questionnaire and 0.40 using the job specific questionnaire for toluene/xylene, 0.21 and 0.28 respectively for chromium
- Specificities = 0.92 and 0.93 for toluene/xylene, 0.94 and 0.93 for chromium
- Kappa in comparison to urine samples- Kappas = 0.24 and 0.37 for toluene/xylene, 0.18 and 0.26 for chromium
Stewart et al, 20009330 jobs randomly selected from each of 10 formaldehyde and resin manufacturing plants participating in a cohort study in the USThree industrial hygienists assessed exposure to formaldehyde into 4 ordinal categories, in 6 stages using increasing amounts of data, including job, department, industry, date, and plant reportsOriginal exposure estimates developed for the cohort study by 2 industrial hygienists using walkthrough surveys, historical documents, formaldehyde measurement data, and interviews with long-term employees, process flow and change data, exposure controls- Relative bias and relative standard deviation, compared to mean exposure estimates- Mean relative bias of −0.12 and relative standard deviation of 0.54 with the first stage of information
- Mean relative bias less than ±0.05 and relative standard deviation less than 0.5 in the remaining 5 stages of data
- Intraclass correlation coefficients for inter-rater agreement- Intraclass correlations ranged from 0.42 to 0.51, with no clear pattern according to amount of data provided