Article Text

Download PDFPDF

Original article
Rule-based exposure assessment versus case-by-case expert assessment using the same information in a community-based study
  1. Susan Peters1,
  2. Deborah C Glass2,
  3. Elizabeth Milne3,
  4. Lin Fritschi1,
  5. the Aus-ALL consortium
  1. 1Epidemiology Group, Centre for Medical Research, University of Western Australia, Perth, Western Australia, Australia
  2. 2Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
  3. 3Telethon Institute for Child Health Research, University of Western Australia, Perth, Western Australia, Australia
  1. Correspondence to Dr Susan Peters, Epidemiology Group, Centre for Medical Research, University of Western Australia, 35 Stirling Hwy, Perth, WA 6009, Australia; susan.peters{at}uwa.edu.au

Abstract

Background Retrospective exposure assessment in community-based studies is largely reliant on questionnaire information. Expert assessment is often used to assess lifetime occupational exposures, but these assessments generally lack transparency and are very time-consuming. We explored the agreement between a rule-based assessment approach and case-by-case expert assessment of occupational exposures in a community-based study.

Methods We used data from a case–control study of childhood acute lymphoblastic leukaemia in which parental occupational exposures were originally assigned by expert assessment. Key questions were identified from the completed parent questionnaires and, on the basis of these, rules were written to assign exposure levels to diesel exhaust, pesticides and solvents. We estimated exposure prevalence separately for fathers and mothers, and used κ statistics to assess the agreement between the two exposure assessment methods.

Results Exposures were assigned to 5829 jobs among 1079 men and 6189 jobs among 1234 women. For both sexes, agreement was good for the two assessment methods of exposure to diesel exhaust at a job level (κ=0.70 for men and κ=0.71 for women) and at a person level (κ=0.74 and κ=0.75). The agreement was good to excellent for pesticide exposure among men (κ=0.74 for jobs and κ=0.84 at a person level) and women (κ=0.68 and κ=0.71 at a job and person level, respectively). Moderate to good agreement was observed for assessment of solvent exposure, which was better for women than men.

Conclusions The rule-based assessment approach appeared to be an efficient alternative for assigning occupational exposures in a community-based study for a selection of occupational exposures.

Statistics from Altmetric.com

What this paper adds

  • Retrospective exposure assessment in community-based studies is often performed by experts on a case-by-case basis, but these assessments generally lack transparency and are very time-consuming.

  • We explored the agreement between a standardised rule-based assessment approach and case-by-case expert assessment of three occupational exposures that are different in occurrence and distribution over workplaces.

  • The rule-based approach appeared to be an efficient alternative for assigning occupational exposures to diesel exhaust and pesticides, and to a lesser extent for solvents, in a community-based case–control study. This method will be particularly useful for risk identification.

Introduction

Retrospective exposure assessment in community-based studies is largely reliant on questionnaire information. Case-by-case expert assessment is often used to assess lifetime occupational exposures and is considered to be a reliable method.1 ,2 However, these assessments generally lack transparency and are very time-consuming.

Recently, several developments to structure expert assessment in community-based studies have been described. Fritschi and colleagues developed a web-based application (OccIDEAS) to automate part of the expert assessment process.3 This application uses predefined exposure assignment rules and makes the process more transparent and objective, and, although experts still need to make some of the final decisions, improves the efficiency.3 Pronk et al4 applied algorithms to assign decision rules to assess occupational exposure to diesel exhaust in a US case–control study of bladder cancer. Moderately high agreement was observed between the algorithm and expert assessment, showing that such a framework may improve the efficiency, consistency and transparency of the exposure assessment process. However, the authors advised caution because assessment of diesel exhaust exposure is relatively straightforward, and therefore the observed advantages might not be generalisable to other occupational exposures.4 OccIDEAS and the algorithm approach both provided an instrument to replicate exposure decisions in other studies.3 ,4

The Australian acute lymphoblastic leukaemia (ALL) study was a case–control study that used case-by-case expert assessment to investigate the association between parental occupational exposures and childhood ALL. Results have been published previously.5–7 The questionnaire information became available for exposure assessment in six batches as they were collected. It was a time-consuming process to individually assess the thousands of job histories, which also made consistency over time difficult. Another study on childhood brain tumours (Australian Childhood Brain Tumour Study (Aus-CBT)) was recently conducted with the same study design and data collection instruments.8 In a quest to identify a more standardised and, above all, efficient way to accurately assess occupational exposures in Aus-CBT, we explored the possibility of applying a ‘rule-based assessment’ to replace the case-by-case expert assessment.9 ,10 The Aus-ALL study provided us with the opportunity to compare the two methods and to investigate the feasibility of a rule-based approach among both men and women in a community-based study.

For the current comparison of assessment methods, we have selected three occupational exposures that are different in occurrence and distribution over workplaces. The first is diesel exhaust, which is a combustion product affecting workers in a wide range of jobs. Second is pesticide exposure, which is related to a relatively limited number of specific tasks in a limited number of occupations. Lastly, we assessed exposure to solvents, as these include aromatic, aliphatic and chlorinated solvents occurring in a variety of workplaces.

Methods

Aus-ALL was a nation-wide Australian case–control study of risk factors for childhood ALL. Full details of the study have been published elsewhere.11 Three controls, frequency matched by age, sex and state of residence, were recruited for each case by random digit dialling. In total, 416 cases and 1361 controls aged under 15 years were recruited from mid-2003 to 2006.

Both parents were asked to complete a written questionnaire requesting demographic and lifestyle details and a lifetime occupational history up to the birth of the child for the father and up to 1 year after the birth for the mother. Information requested about each job included the year started and finished, job title, employer, main tasks, and hours worked each week. If the mother or father reported ever having worked in any of certain jobs (carpenter/cabinet maker, chemist, office worker, drycleaner, driver, engineering technician, farmer, fisherman, gas station attendant, hairdresser, health professional, labourer, mechanic, miner, metal worker, painter, printer, radio operator, railway worker, shoemaker, store man or teacher) or working in particular industries (aluminium, forestry, military, leather, oil refining, rubber or textile), they were telephoned, and trained interviewers used job-specific modules (JSMs) to ask them further detailed questions about tasks they undertook in that job. The relevant JSMs were assigned by members of the Aus-ALL research team before contacting the subjects by telephone.

An occupational hygienist (DCG, 2005–2008) reviewed all job histories to assess exposure to solvents, exhausts, paints and pigments, glues, pesticides, lead, ionising and non-ionising radiation, and extremely low frequency electromagnetic fields. Blinded to case status, she determined the likelihood of exposure (no, possible or probable), level of exposure (low, medium or high) and frequency of exposure (hours per week and weeks per year). These assessments have been used in previous reports from the Aus-ALL study.5–7 Occupational histories and corresponding case-by-case expert assessments were available for 1079 men (328 case and 751 control fathers) and 1234 women (379 case and 855 control mothers). For the current analyses, only likelihood of exposure was taken into account, where jobs were considered exposed when classified by the expert as ‘probable’ exposed. Jobs with ‘no’ or ‘possible’ exposure were considered non-exposed.

For the ‘rule-based assessment’, one of the authors (SP, 2012) selected key questions that indicated exposure from the JSM interviews. On the basis of the answers to these questions, rules were written to assign exposures (yes/no) to diesel exhaust, pesticides and solvents (benzene, other aromatics, aliphatics and chlorinated solvents). The preliminary rules were discussed with the exposure expert who performed the original exposures (DCG) and were modified where necessary to finalise the rules to be used in the present comparison study.

The rules we developed were based on the questionnaire used for this study. For the different exposures, different numbers of questions were selected: for pesticides, the assessment was based on 11 specific questions from six JSMs; diesel exhaust was based on 26 questions from 11 JSMs; benzene was based on 16 questions from 14 JSMs; other aromatic solvents were based on 31 questions from 16 JSMs; aliphatic solvents were based on 30 questions from 16 JSMs; and chlorinated solvents were based on 20 questions from 11 JSMs. Rules that led to assignment of exposure to an agent were either based on the response to one question only (eg, exposure to diesel exhaust when the answer to ‘What kind of vehicle did you usually drive?’ was ‘a bus’ or ‘a truck’), or based on the combination of responses to two or three questions. An example of the latter is assigning exposure to diesel exhaust when ‘yes’ was answered to both the questions ‘Were trucks being loaded or unloaded in the area where you worked?’ and ‘Were their engines usually running?’. Time period was taken into account for use of chlorinated solvents in the printing industry, for which exposure was only assigned when the job was before 1990.

In all, 9.5% (n=103) of the fathers and 18.7% (n=231) of the mothers did not report working in any of the relevant jobs or industries and were therefore not assigned an interview. JSM interviews were conducted with 97.9% (n=955) of the fathers and 96.8% (n=970) of the mothers who had reported working in at least one of the relevant jobs or industries. Not all JSMs of the relevant jobs were included in the interview, however, because of the burden on the respondent. If more than five JSMs were assigned to a subject, a selection of five JSMs was made by the research team before the interview; these represented the longest held, or the most different from other jobs. In total, interviews were available for 62.1% (n=2416) of the JSMs for relevant jobs of the fathers and for 60.8% (n=2053) of those of the mothers. If no JSM was available for a parent, the original exposure assessment was based on information from similar JSMs answered by that parent. If no similar JSMs were completed, a generic assessment was assigned on the basis of typical answers from other parents with that job. The rule-based approach used the originally assigned JSM to assess the exposures. A JSM without interview was then assessed as ‘exposed’ if, on the basis of the interviews held in the rest of the study population, an exposure was assigned to more than 50% of that particular JSM.

We compared the original case-by-case expert assessment with the rule-based assessment. Both methods were reliant on the same digitised information as extracted from the telephone interviews using JSMs. We calculated κ statistics as a measure of inter-method agreement.12 Assignments using the original case-by-case expert assessment were compared with those using the rule-based approach, based on exposed versus non-exposed jobs and persons. κ was interpreted using the following arbitrary cut-off points: <0.4, poor; 0.4–0.75, moderate to good; >0.75, excellent.12 Statistical analyses were conducted using SAS V.9.3.

Results

Exposures were assessed for 5829 jobs among 1079 men and 6189 jobs among 1234 women from the Aus-ALL study. Overall, exposure prevalence was lower among women than men (table 1). For both sexes, agreement was good for the two assessment methods of exposure to diesel exhaust at a job level (κ=0.70 for men and κ=0.71 for women) and at a person level (κ=0.74 and κ=0.75). The agreement was good to excellent for pesticide exposure among men (κ=0.74 for jobs and κ=0.84 at a person level), and assessments for pesticide exposure among women showed good agreement (κ=0.68 and κ=0.71 at a job and person level, respectively).

Table 1

Comparison between exposure assessment methods for three occupational exposures

Overall, better agreement was observed for assessments of solvent exposure for women than men. Agreements between the two assessment methods for the specific solvents were moderate to good for benzene, other aromatics and aliphatics in both men and women (table 2). The agreement was lowest for chlorinated solvents (κ=0.26 at a job level for men, and κ=0.46 for women).

Table 2

Comparison between exposure assessment methods for occupational exposures to specific solvents

Table 3 shows the agreement between the two methods separately for jobs where interview data were available and for jobs without an interview. The latter group also included jobs where no JSM was assigned. Better agreement was observed for jobs with interview data (ranging from κ=0.56 to κ=0.80) than for jobs for which no interview was performed (κ=0.31–0.54).

Table 3

JSM agreement at job level between the two exposure assessment methods for three occupational exposures, stratified by availability of interview data

Discussion

Rule-based exposure assessment has the potential to replace case-by-case expert assessment of occupational exposures in community-based studies that have to rely on questionnaire data. The main advantages of a rule-based approach are: (a) full standardisation of the exposure assessment process; (b) documentation of all decisions made in the assessment, which increases the transparency; and (c) a less labour-intensive method, which will save time and expense.

Overall agreement between the exposures assessed by algorithm and by expert was good at both a job and a person level. Neither of the two methods appeared to assign systematically a higher or lower prevalence of exposure than the other. We collected occupational histories from both parents, so we were able to compare the results among men and women. Since men are generally more often employed in jobs with exposures to hazardous substances, it was not surprising that exposure prevalence was higher for men for all agents assessed. Agreements between the two assessment methods were comparable between men and women. Only the agreements for exposure to solvents were somewhat higher among women.

So far, the rule-based approach has only been described and evaluated for diesel exhaust exposure in a US case–control study of bladder cancer.4 The researchers observed high agreement (κweighted=0.81) between the algorithm and expert assessment method for probability of diesel exhaust exposure at a job level, which they defined in four categories.4 This agreement is comparable to our findings: κ=0.70 for men and κ=0.71 for women. Assessment of exposure to diesel exhaust is considered relatively straightforward: diesel exhaust has an offensive odour, and the source (diesel engines in vehicles or other equipment) is clear. The accuracy of assessment of exposure to diesel exhaust may therefore be greater than for other agents.4

In our study, however, exposure to pesticides showed the highest agreement (κ = 0.74 at a job level for men). This high agreement is probably due to the clear tasks related to pesticide exposure. Agreements for solvent exposures were somewhat lower, but still moderate to good in most cases, except for chlorinated solvents (κ=0.26 for men and κ=0.46 for women at a job level). Chlorinated solvents appear to be particularly difficult to assess. For example, a study comparing the assessments by experts from different study centres showed poor agreement for chlorinated solvents (κ=0.11), while agreement was excellent for diesel exhaust (κ=0.80).13 A recent study on trichloroethylene exposure and non-Hodgkin lymphoma used a comprehensive exposure assessment method, combining occupational histories, job- and industry-specific modules with specific focus on solvents, and task-, job-, industry- and decade-specific exposure matrices developed on the basis of an extensive literature review.14 The industrial hygiene expert considered all these data when assigning potential exposure to trichloroethylene. The use of more detailed information for assessment of exposure to trichloroethylene enabled the detection of significantly increased risks of non-Hodgkin lymphomas, which were not detected using less detailed exposure assessment methods.14

The aim of the rule-based assessment was to standardise the exposure assessment process. Although the approach is different from the case-by-case expert assessment, their starting points are the same. Both methods rely on the same information from the subjects (ie, responses to questionnaires) and expert knowledge, although in the case-by-case assessment an expert is able to take into account free-text fields as well. On the one hand, differences between the rule-based approach and the expert assessment may reveal the lack of flexibility of the rules in cases where additional information leads to the classification of exposure by the expert. OccIDEAS, for example, therefore uses expert views as a complement to the rules for complex decisions.3 Agreement between the two methods in our study was higher when we limited the comparison to the jobs for which interview data were available (table 3). This observation may indicate that, in situations where interview data are not available, individual assessments based on free-text fields on job, employer and tasks may be required. On the other hand, differences between the methods may reflect the main weakness of case-by-case expert assessment, namely that there is inevitably a degree of subjectivity. The process of assessing thousands of job histories and assessing exposures to numerous agents takes several months, which makes it nearly impossible to be perfectly consistent.

In the US study, statistical learning techniques (ie, classification and regression tree (CART) and random forest models) were applied to identify the underlying rules of the expert's exposure assignments to diesel exhaust.15 In that study, two exhaust-specific questions were available for each job of each participant (ie, ‘While at this job, did you ever work near diesel engines or other types of engines?’ and ‘While at this job, did you ever smell diesel exhaust or other types of engine exhaust?’).4 The variable constructed from these two questions appeared to be the most predictive.15 However, this type of information is generally not available for all subjects in community-based studies. This was also not the case in our study, where there was a wide range of questions asked in the different JSMs. Therefore, we identified the key questions for each exposure manually. Identification of the most important information for an exposure decision will help in refining questionnaires in future studies.15 This can reduce the respondent burden, which may improve the feasibility of data collection in case–control studies, as time needed for occupational data collection can be shortened by asking only essential questions.

We have only compared the assessment of exposed versus non-exposed, which is sufficient for risk identification. For risk quantification, however, more detailed information is required. Measurement data would be preferred, as quantitative exposure assessment is the ultimate goal in all occupational studies.16–18 Unfortunately, measurements are not always available, particularly not in community-based case–control studies, which are the most efficient type of study for investigating risk factors for rare diseases. In those situations, input from experts can be used to assign exposure levels.19 Besides a rule-based approach, the use of job-exposure matrices is another standardised and transparent method that is often applied in community-based studies.20 ,21 Unlike job-exposure matrices, however, rule-based assessment based on questionnaire information allows account to be taken of individual differences between workers within the same job title.22 The main limitation of the rule-based approach, as well as of case-by-case expert assessment, is that it is dependent on the questions selected for the respondents and their answers. This might lead to misclassification because of missing information or recall bias.

To determine the implications of the differences between methods, resulting risk estimates should be compared. Ideally this would be carried out in a study with an established association between occupational exposure and disease outcome, as, for example, has been done for several lung carcinogens.21 The ability of the method to detect a known association gives an indication of its effectiveness. No external exposure has yet been established as a cause of childhood ALL, so we were not able to test the effectiveness of the methods in this way. Nevertheless, the κ values presented in this paper show a moderate to good agreement between the rule-based approach and the conventional method.

Overall, the rule-based exposure assessment approach appeared to be an efficient way to assign occupational exposures to diesel exhaust and pesticides, and to a lesser extent for solvents, in a community-based case–control study. This method may therefore be applicable as a substitute for case-by-case expert assessment, particularly for risk identification. While both methods use the same information, the rule-based approach is more standardised and transparent and less time-consuming.

Acknowledgments

The Aus-ALL consortium conducted the study, and the Telethon Institute for Child Health Research (TICHR), University of Western Australia, was the coordinating centre. Bruce Armstrong (Sydney School of Public Health), EM (TICHR), Frank van Bockxmeer (Royal Perth Hospital), Michelle Haber (Children's Cancer Institute Australia), Rodney Scott (University of Newcastle), John Attia (University of Newcastle), Murray Norris (Children's Cancer Institute Australia), Carol Bower (TICHR), Nicholas de Klerk (TICHR), LF (WA Institute for Medical Research), Ursula Kees (TICHR), Margaret Miller (Edith Cowan University) and Judith Thompson (WA Cancer Registry) were the research investigators, and Helen Bailey (TICHR) was the project coordinator. The clinical investigators were: Frank Alvaro (John Hunter Hospital, Newcastle); Catherine Cole (Princess Margaret Hospital for Children, Perth); Luciano Dalla Pozza (Children's Hospital at Westmead, Sydney); John Daubenton (Royal Hobart Hospital, Hobart); Peter Downie (Monash Medical Centre, Melbourne); Liane Lockwood, (Royal Children's Hospital, Brisbane); Maria Kirby (Women's and Children's Hospital, Adelaide); Glenn Marshall (Sydney Children's Hospital, Sydney); Elizabeth Smibert (Royal Children's Hospital, Melbourne); Ram Suppiah (previously Mater Children's Hospital, Brisbane).

References

Footnotes

  • Collaborators Bruce Armstrong; Frank van Bockxmeer; Michelle Haber; Rodney Scott; John Attia; Murray Norris; Carol Bower; Nicholas de Klerk; Ursula Kees; Margaret Miller; Judith Thompson; Helen Bailey; Frank Alvaro; Catherine Cole; Luciano Dalla Pozza; John Daubenton; Peter Downie; Liane Lockwood; Maria Kirby; Glenn Marshall; Elizabeth Smibert; Ram Suppiah.

  • Contributors SP and DCG were responsible for the exposure assessment. SP and LF were responsible for the statistical analysis and the interpretation of data. EM was the principle investigator of the Aus-ALL study. The paper was drafted by SP and was revised with contributions from all coauthors. All authors reviewed and approved the paper.

  • Funding Aus-ALL was funded by a National Health and Medical Research Council (NHMRC) Grant (254539). LF holds fellowships from the NHMRC and Cancer Council WA.

  • Competing interests None.

  • Patient consent Obtained.

  • Ethics approval Human research ethics committee approval was obtained from participating hospitals.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.