Validity and reliability of exposures estimated by experts
| Authors, year | Study population | Expert measure | Comparison measure | Method of assessing validity or reliability | Results |
|---|---|---|---|---|---|
| Woitowitz et al, 197070 | Employees of a raw asbestos processing plant in Germany | Study personnel ranked employees' asbestos exposure on a 4-point scale, based on tours of the worksite, and consultations with department heads, trade union personnel, plant or government physician, firm safety officer, and shop committee | 61 dust measurements | - Comparisons of mean dust concentrations of expert-assessed exposure categories | - Mean dust concentration for category 1 = 0.6 mg/m3; for category 2 = 1.2 mg/m3; for category 3 = 1.6 mg/m3 |
| Goldberg et al, 198678 | Subjects of a case-control study among Canadian men with cancer at any of 19 tumor sites | A rater (occupational hygienist, chemist, or engineer) assessed subjects' presence or absence of exposures to 172 to 275 substances, based on information provided by subjects in a detailed interview, their own knowledge, review of bibliographic materials, and consultations with other experts | Assessments of 1 to 2 other raters on the study team using the same methods; and assessments by an expert from certain industries external to the study | - % agreement between raters; kappa | - Average % agreement in 6 inter-rater agreement trials ranged from 95.5% to 98.5% |
| - Average kappas ranged from 0.51 to 0.67 | |||||
| Kromhout et al, 198747 | Employees of a paint factory (n=29), a food processing facility (n=58), a nonwoven materials factory (n=164), and 2 coach works (n=144) in the Netherlands | Two occupational hygienists ranked employees' exposures to either dust or solvents using a 4-point scale | 58 solvent measurements in the paint factory; and 421 dust measurements in the other plants | - Proportion of variance in exposure explained (adjusted R2) by the hygienists' rankings | - R2 ranged from 0.08 to 0.27 for dusts, median = 0.25 |
| - R2 were 0.37 and 0.58 for the solvents, for the two hygienists | |||||
| Ciccone and Vineis, 198879 | 88 soft tissue sarcoma cases and 157 population based controls from a rice growing region of Italy | An agricultural chemist with rice growing expertise assessed subjects' exposure to phenoxy herbicides using a 3-point scale, using information collected from the subjects on their job history, locations of farms, types of crops, characteristics of pesticide handling, and their subjective estimate of pesticide exposure | Assessments by a similarly expert agricultural chemist, using the same method | - % agreement between chemists; kappa | - 95.5% agreement |
| - weighted kappa = 0.76 | |||||
| Hawkins and Evans, 198980 | 12 chemical process workers in the US | 24 occupational hygienists with expertise in chemical processing randomly selected from 2 professional organizations gave quantitative estimates of the employees' toluene exposures, first based on a qualitatitive description of the process and work environment, then after viewing limited historical measurement data | 134 toluene exposure measurements | - Comparison of mean, median, and 90th percentiles of exposure estimates to measured exposures | - Mean measured exposure = 4.6 ppm; first estimate = 14 ppm; second estimate = 4.4 ppm |
| - Median measured exposure = 0.37 ppm; first estimate = 8.9 ppm; second estimate = 3.0 ppm | |||||
| - 90th %ile of measured exposure = 16 ppm; first estimate = 41 ppm; second estimate = 21 ppm | |||||
| Teschke et al, 198951 | 225 Canadian sawmill workers | Three pairs of occupational hygienists estimated hours of exposure to chlorophenate fungicides per year, after walk through survey of sawmill | Concentration of total chlorophenate in urine samples, measured in 150 workers in the summer, and 154 workers in the fall | - Proportion of variance in urinary chlorophenate concentrations explained by hygienists' estimates of hours of exposure (R2) | - Lumber industry hygienists: R2 = 0.08; with skin exposure also included in the model, R2 = 0.26; government hygienists: R2 = 0.24; other industry hygienists: R2 = 0.22 |
| - Intraclass correlation coefficients for agreement between members of each pair of hygienists | - Lumber industry hygienists: ICC = 0.68; government hygienists: ICC = 0.40; other industry hygienists: ICC = 0.50 | ||||
| Post et al, 199181 | Employees in nine jobs in a small polyester factory in the Netherlands | 9 occupational hygienists each classified employees' exposures to styrene and methylene chloride into three absolute categories related to the Threshold Limit Value, using information about the process and jobs, and a visit to the plant; estimation was done two additional times after receipt of a small amount of measurement data | 45 styrene and 28 methylene chloride exposure measurements | - Spearman rank correlation coefficient comparing relative ranking of jobs by each hygienist to measured ranking | - Spearman r ranged from 0.3 to 0.9 for methylene chloride, median = 0.65; and from −0.4 to 0.65 for styrene, median = 0.2 |
| - % agreement between absolute classifications of jobs by each hygienist and measured classifications | - % agreement ranged from 0.15 to 1.0 for methylene chloride, median = 50%; and from 0.1 to 1.0 for styrene, median = 55% | ||||
| Dovan et al, 199382 | 81 homes of cases and controls in a study of childhood cancer in the US | A study technician trained in a “wire coding” procedure classified the current configuration of the electrical transmission and distribution lines near each home into one of four ordinal categories in 1985 | - Repeat of the wire coding in 1990 | - % agreement in wire codes | - 90% agreement |
| Macaluso et al, 199383 | 29 paint department employees of a car assembly plant in the US | Occupational hygienist (university or consultant) classified exposures of 695 job-department-year combinations to 6 product use groups and 7 specific chemical groups, into six absolute concentration categories, based on historical measurement data | Assessments by 4 similar hygienists, using the same method | - Intraclass correlation coefficient for all job-department-year exposure scores and for cumulative exposure scores | - ICCs for all job-department-year exposure scores ranged from −0.05 to 0.64, median = 0.14; ICCs for cumulative exposures from 0 to 0.85, median = 0.16 |
| - ICCs for all job-department-year exposure scores ranged from −0.07 to 0.33, median = 0.24; ICCs for cumulative exposures from 0 to 0.58, median = 0.33 | |||||
| Takahashi et al, 199484 | 42 deceased Canadian men who had cancer at various sites, all subjects in a case-control study | A team of experts (occupational hygienists, chemists, or engineers) ranked subjects' exposures to asbestos into 3 ordinal categories based on information provided by subjects in a detailed interview, their own knowledge, review of bibliographic materials, and consultations with other experts | Measured asbestos fibre concentrations in 42 lung tissue samples taken at autopsy | - Comparisons of mean lung asbestos concentrations for expert assessed exposure categories | - Mean fibre concentration of subjects rated as having no asbestos exposure history = 0.09 f/ug dry lung; with low or moderate estimated exposure concentration = 0.14 f/ug; with high estimated exposure concentration = 8.7 f/ug |
| - Proportion of variance in lung asbestos concentration explained (R2) by experts' ratings | - R2 = 0.32, with age also in model, p<0.0006 | ||||
| Armstrong et al, 199685 | 31 cases of lymphohaematopoetic cancer and 124 controls who were subjects of a nested case-control study of petroleum marketing and distribution workers in Canada | Study experts estimated exposures to total hydrocarbons and benzene, based on an algorithm which included mean measured exposure levels, and modifying information about the workplace, tasks performed, ambient environment, and products | 15 measurements of total hydrocarbon exposure in the 6 job-years, and 51 measurements of benzene exposure in the 9 job-years; these data were withheld from the expert estimation process | - % difference in mean algorithm estimated exposures to withheld measurement means | - % difference for total hydrocarbons ranged from −49% to 220%, median = −35% |
| - % difference for benzene ranged from −14% to 130%, median = 4.5% | |||||
| de Cock et al, 199686 | 15 fruit growing farms in the Netherlands | 15 occupational hygienists, pesticide experts, and fruit growers ranked potential for pesticide exposure by dermal and inhalation routes, of 14 tasks and of 15 spraying activities; done in three phases: after viewing a video on factors affecting exposure, after viewing slides about pesticide application tasks, and after reading written information on tasks and pesticides | Measurements of airborne and dermal captan concentrations during spraying | - Spearman rank correlation coefficients comparing relative ranking of activities by each expert to measured ranking | - Spearman r ranged from −0.1 to 0.45 for inhalation exposure, median = 0.3; and from 0.03 to 0.9 for dermal exposure, median = 0.65 |
| - Intraclass correlation coefficients for agreement between experts | - ICCs ranged from 0.61 to 0.81 for inhalation exposure, median = 0.72; and from 0.53 to 0.71 for dermal exposure, median = 0.63 | ||||
| Segnan et al, 199687 | 82 vineyard workers, 171 metal plating workers, and 158 leather goods workers in Italy | Using occupational histories, industry specific questionnaires, lists of products used, and where available, exposure measurement data (in separate stages) | The same experts' ratings using the full set of data available to them | - Sensitivity and specificity using the ratings based on the full datatset as the gold standard | - Sensitivities ranged from 0.13 to 0.99 (median = 0.78) using only the occupational histories, 0.05 to 1.0 (median = 0.76) using the industry-specific questionnaires, 0.87 to 1.0 (median = 0.97) using the product lists |
| - 8 agronomists assessed exposure to 10 pesticides, 6 classes of chemicals, and to the broad groups “fungicides” and “insecticides” | - Specificities ranged from 0.12 to 0.90 (median = 0.52) using only the occupational histories, 0.17 to 0.99 (median = 0.77) using the industry-specific questionnaires, 0.77 to 1.0 (median = 1.0) using the product lists | ||||
| - 8 industrial hygienists assessed exposures to 20 chemicals and 5 classes of chemicals | |||||
| - 4 industrial hygienists assessed exposures to 20 solvents and 9 classes of chemicals | |||||
| - Intraclass correlation coefficients for the agreement between raters | - ICCs ranged from below 0 to 1.0 for all methods, with a median of 0.11 using only the occupational histories, 0.21 using the industry-specific questionnaires, 0.65 using the product lists, and 0.51 using monitoring data | ||||
| Benke et al, 199788 | Jobs reported in a case-control study of glioma in Australia | 3 industrial hygienists and 2 occupational physicians assessed the presence or absence of exposure to 21 chemicals in 199 jobs randomly selected from subjects' histories and 49 jobs with exposure measurements | Industrial hygiene reports for the 49 jobs from a database of surveys in the study region, over the period from 1978–1989 | - Sensitivity and specificity for exposure in 49 jobs with exposure measurements | - Sensitivities ranged from 0.48 to 0.79, median = 0.65; specificities ranged from 0.91 to 0.98, median = 0.94 |
| - Kappas for pairwise inter-rater agreement for the 199 jobs of study subjects | - Kappas for inter-rater agreement ranged from 0 to 0.64, median = 0.19 | ||||
| - Kappas for intra-rater agreement based on reassessment of 50 of the 199 jobs, at least 4 months later | - Kappas for intra-rater agreement ranged from 0.46 to 0.73, median = 0.60 | ||||
| McGuire et al, 199789 | 179 job histories from population based case-control study of amyotrophic lateral sclerosis in the US | Panel of four industrial hygienists, by consensus, rated exposure to three groups of chemical agents based on job history information | Blinded repeat assessment by same panel | - % agreement and kappas | - % agreement 90% for metals, 82% for solvents, and 97% for agricultural chemicals |
| - Kappas were 0.77 for metals, 0.64 for solvents, and 0.75 for agricultural chemicals | |||||
| Semiatycki et al, 199790 | 50 subjects from a case-control study among Canadian men with cancer at any of 19 tumour sites | Consensus review by two or more experts (industrial hygienists or chemists) of subjects' histories of their jobs, work environments, raw materials, products, and self reported exposures to assess presence and probability of exposure to 294 chemical and physical agents | Two experts reassessed exposure to 94 jobs by consensus, and reassessed exposure to 92 different jobs independently from one another | - Kappas | - In consensus reassessment, weighted kappa for 4 categories of probability of exposure = 0.80; kappas for presence of exposure ranged from 0.51 to 0.94 for the 18 highest-prevalence substances, median = 0.75 |
| - In independent reassessments, weighted kappas for 4 categories of probability of exposure ranged from were 0.73 and 0.76 for the two experts | |||||
| Rybicki et al, 199891 | Job histories of 60 and 64 of 608 subjects in a case-control study of neurologic disease in the US | Expert review by two industrial hygienists of self reported exposures to copper, lead and iron, of 60 and 64 study subjects | Prior expert review by one of the same industrial hygienists of self reported exposures of all 608 study subjects | - Kappas for intra-rater and inter-rater agreement | - Kappas = 0.26 for copper, 0.56 for lead, and 0.57 for iron, intra-rater |
| - Kappas = 0.15 for copper, 0.29 for lead, and 0.49 for iron, inter-rater | |||||
| Cherrie and Schneider, 199992 | 17 jobs in brick manufacturing in the UK, 13 jobs in rubber and pigment coating in the UK, 14 jobs in fibre reinforced plastics in Denmark, 13 jobs in an asbestos contaminated warehouse in the US, and 6 jobs in man made mineral fibre manufacturing in the US | Two industrial hygienists' estimates of exposure concentrations of respirable dust, toluene, styrene, asbestos, or man made mineral fibre, based on descriptions of the jobs, tasks, work environments, and control measures, using a structured assessment method based on emissions, processing at the source, and controls | Measurements of airborne concentrations of respirable dust, toluene, styrene, asbestos, and man made mineral fibre | - Correlation coefficient comparing log transformed exposure measurements to the hygienists' estimates | - Pearson r ranged from 0.31 to 0.93 for all agents except styrene, and from 0 to 0.31 for styrene, for which there was little variability in measured exposure, median = 0.39 |
| - Bias, as measured by ratio of the geometric mean exposure estimate to the geometric mean measured concentration | - Bias ratio ranged from 0.47 (underestimate of exposure) to 2.86 (overestimate), median = 1.47 | ||||
| Tielemans et al, 199943 | Subjects of 2 case-control studies of male infertility in the Netherlands | Researchers used subject reported data from generic job histories and job specific questionnaires (additional data on tasks and substances used) to assess exposures to solvents and chromium | - Urine samples analysed for metabolites of toluene and xylene (n=267) and for chromium (n=156) | - Sensitivity and specificity in comparison to urine samples | - Sensitivities = 0.30 using the generic questionnaire and 0.40 using the job specific questionnaire for toluene/xylene, 0.21 and 0.28 respectively for chromium |
| - Specificities = 0.92 and 0.93 for toluene/xylene, 0.94 and 0.93 for chromium | |||||
| - Kappa in comparison to urine samples | - Kappas = 0.24 and 0.37 for toluene/xylene, 0.18 and 0.26 for chromium | ||||
| Stewart et al, 200093 | 30 jobs randomly selected from each of 10 formaldehyde and resin manufacturing plants participating in a cohort study in the US | Three industrial hygienists assessed exposure to formaldehyde into 4 ordinal categories, in 6 stages using increasing amounts of data, including job, department, industry, date, and plant reports | Original exposure estimates developed for the cohort study by 2 industrial hygienists using walkthrough surveys, historical documents, formaldehyde measurement data, and interviews with long-term employees, process flow and change data, exposure controls | - Relative bias and relative standard deviation, compared to mean exposure estimates | - Mean relative bias of −0.12 and relative standard deviation of 0.54 with the first stage of information |
| - Mean relative bias less than ±0.05 and relative standard deviation less than 0.5 in the remaining 5 stages of data | |||||
| - Intraclass correlation coefficients for inter-rater agreement | - Intraclass correlations ranged from 0.42 to 0.51, with no clear pattern according to amount of data provided |