Background: Occupational hygiene panels are increasingly being used to rate retrospective occupational exposures to chemicals in community-based studies. This study aimed to assess the validity, reliability and feasibility of using such an expert panel in a brain tumour case-control study.
Methods: A panel of five experts was recruited to rate exposure to 21 chemicals for 298 job descriptions to investigate the level of agreement. Validity was assessed by comparing the ratings of the experts for 49 of the jobs with objective quantitative exposure data which existed for these jobs. Repeatability was assessed by comparing the results for 50 resubmissions.
Results: Specificity was high for reporting that exposure occurred (all above 90%), but sensitivity was variable with values between 48% and 79%. Weaker validity was found for rating exposure level and exposure frequency. The raters showed the greatest inter-rater agreement for exposure to three of the 21 chemicals considered (kappa = 0.64 for cutting fluids, kappa = 0.57 for welding fumes and kappa = 0.42 for lubricating oils). Intra-rater reliability, based on the 50 resubmitted jobs, was fair to good (kappa = 0.46, 0.73).
Conclusions: The potential effect of exposure misclassification from using expert panels was quantified and found to be a significant source of bias. The optimum situation occurred where three of the five raters concurred, where an odds ratio of 2.2 was observed for a true odds ratio of 4.0. Future studies which plan to use expert panels should screen the experts for their suitability by validating their performance against jobs with known exposure data.