Background/aim: This study used information from the questionnaire alone or in conjunction with clinical tests, such as skin-prick testing (SPT) and bronchial responsiveness (BR) testing at entry, to develop models for estimating the probability of the occurrence of specific IgE-sensitisation to and respiratory symptoms in contact with laboratory animal (LA) allergens after 32 months’ training in an animal health technology programme.
Methods: Four multivariable logistic regression models were developed for each endpoint, consisting of: (1) questionnaire; (2) questionnaire and SPT; (3) questionnaire and BR testing; and (4) questionnaire, SPT and BR testing. The prognostic models were derived from a cohort of Canadian animal health technology apprentices. The models’ internal validity and diagnostic accuracy were evaluated and compared.
Results: Symptoms indicative of asthma and allergic symptoms at baseline composed the final questionnaire model for the occurrence of occupational sensitisation and symptoms. Both questionnaire models showed a good discrimination (area under the receiver operating characteristics curve were 0.73 and 0.78, respectively) and calibration (Hosmer-Lemeshow test p value >0.10). Addition of SPT and/or BR testing increased the specificity of the questionnaire model for LA sensitisation, but not for symptoms at work. To facilitate their application in practice, the final questionnaire models were converted to easy-to-use scoring system.
Conclusions: Questionnaire is an easy tool that can give accurate prediction of the incidence of occupational sensitisation and symptoms.
Statistics from Altmetric.com
The attributable risk from occupation for asthma in population-based studies ranges from 5% to 15%,1–6 which makes work-related asthma an important public health concern. Several tools are used in the diagnosis of occupational asthma (OA), defined as a type of work-related asthma specifically caused by an agent present at the workplace.7 These include a questionnaire, skin-prick testing, assessment of non-specific bronchial responsiveness, examination of induced sputum, serial monitoring of peak expiratory flows and specific inhalation challenges.7 Where these approaches are used in clinical settings for diagnostic purposes, little is known about their prognostic value in predicting the future occurrence of occupational allergic diseases in open populations, either in general or at specific workplaces. If surveillance or even screening is to be applied in high-risk workplaces, which tools should be used when the worker is hired: a questionnaire alone or its combination with information on atopic status and bronchial responsiveness?
Recently, a multivariable prognostic model for sensitisation to laboratory animal (LA) allergens was proposed.8 The authors used information carried by simple questionnaire items independently or in combination with atopic status at baseline to estimate the probability of the occurrence of occupational sensitisation in three years of follow-up. Nevertheless, the predictive value of these tools with regard to the occurrence of respiratory symptoms at work has not yet been assessed.
Since 1991, we have conducted prospective studies in apprentices exposed to various occupational agents, including high-molecular-weight products—for example, animal-derived allergens, latex and cereals.9–12 Our purpose was to investigate the natural history of occupational allergy and asthma. Students in an animal health technology programme showed a high incidence (8.9 per 100 person-years of sensitisation to LA allergens.10 This group also showed an incidence of probable OA of 2.7 per 100 person-years.11
We used information from a questionnaire alone or in conjunction with clinical tests, such as skin-prick testing (SPT) and bronchial responsiveness (BR) testing at entry, to develop models for estimating the probability of the occurrence of specific IgE-sensitisation to and respiratory symptoms in contact with LA allergens after 32 months training in an animal health technology programme. For their practical application, the models were transformed into easy-to-use score charts. The use of such prognostic models among apprentices exposed to LA allergens may support coaching and job referral, decision-making in medical surveillance and enable early identification of individuals who need support through preventive measures.
Study design and population
The prognostic model was developed in Canadian students in an animal health technology training programme from the cohort apprentice study.10 The apprentices answered questionnaires in a face-to-face interview and were subjected to SPT, lung function and bronchial non-specific provocation tests upon beginning their three-year apprenticeship programme. The tests were subsequently administered yearly until the end of their programme. Informed consent was obtained from each subject and the study was performed after approval from the Sacre-Coeur Hospital’s ethics committee in accordance with Canadian ethical rules.
The number of subjects included in this analysis was different from the one in a previous publication.10 In the previous publication, subjects who showed a sensitisation to LA allergen at the initial visit were also included and considered at risk of developing a new sensitisation to another LA allergen. For the purpose of developing prognostic models for sensitisation to LA allergens, we included only apprentices who had a negative skin reaction to all LA allergens tested at the initial visit; rat urine, mouse urine, rabbit urine and/or rabbit hair. Prognostic models for symptoms at work were derived from apprentices who had neither rhinoconjunctivitis nor chest symptoms at work at the initial visit.
We used questionnaire items, clinical information on atopic status—SPT results to common allergens—and BR to methacholine at baseline as potential prognostic predictors. The questionnaire was derived from the standardised questionnaire of the International Union Against Tuberculosis and Lung Disease (IUATLD) and was administered by a trained nurse, who was blinded to the outcome of SPT and bronchial challenge tests.13 14 We used principal components analysis (PCA) to reduce the number of questionnaire predictors to be included in the multivariate logistic regression analysis.15 PCA is often used to identify underlying variables, or factors, that explain the pattern of correlations within a set of observed variables—for example, questionnaire items on symptoms. From the PCA, we identified clusters of correlated symptoms with Eigenvalues of 1 or more (that is, explaining more variance than a single symptom). Individual symptoms were only included in the clustered variable when factor loadings were higher than 0.4.
There were five clusters of correlated symptoms with Eigenvalues of 1 or more with a cumulative explained variability of 61.1%. The following clusters were identified and named post hoc as adequately as possible: “respiratory symptoms at work”, “symptoms indicative for asthma”, “allergic symptoms”, “symptoms indicative for chronic obstructive pulmonary diseases” and “skin symptoms”. For details regarding symptoms that were included in each cluster see appendix 1 online. A positive cluster was defined as the presence of at least one of the symptoms that composed the cluster.
Methacholine bronchial challenge tests were performed using guidelines slightly modified from those of the European Respiratory Society.16 The procedure for performing the bronchial challenge test was modified as described elsewhere to take into account the absence of an on-site physician.17 BR was considered measurable when the level of the methacholine concentration caused a decrease in forced expiratory volume in one second (FEV1) of 20% (provocative concentration causing a 20% decrease in FEV1, PC20) of 32mg/ml or lower—a measurable PC20. We used a measurable PC20 to define BR because this high cut-off value of PC20 showed the strongest association with the endpoints compared to stricter definitions—for example, PC20 16 mg/ml or lower. SPT based atopy was defined as two positive skin reactions to a set of 11 common inhalants. A wheal diameter of 3 mm or more was regarded as a positive response, in the absence of any reaction to the diluent (glycerine, 50%) and in the presence of a positive reaction to histamine phosphate (1/200 mg/ml).10
Incident cases of sensitisation to LA allergens were defined as individuals with a positive SPT response to rat urine, mouse urine, rabbit urine (Pharmacia Allergon AB, Angelholm, Sweden) and/or rabbit hair (Omega, Montréal, Canada) at the 32nd month visit, but no positive skin reactions to any of these LA allergens tested at the initial visit. A positive skin reaction was defined similarly as to common allergens. Incident cases of symptoms at work were defined as individuals with positive respiratory symptoms at work at the 32nd month visit, but no symptoms at the initial visit.
We developed the prognostic models for sensitisation to the LA allergens in 314 apprentices who had negative skin reactions to any of LA allergens at initial visit. The prognostic models for symptoms at work were developed in 296 apprentices with no work-related symptoms at baseline. We started the multivariable analysis with the newly created clusters from the PCA as well as other personal characteristics with a univariable p value lower than 0.50. We used a backward stepwise selection procedure (using p<0.157 for inclusion) to obtain a questionnaire model with the strongest predictors for sensitisation to LA allergens.18 Subsequently, results from the SPT to common allergens and the bronchial methacholine challenge test were then added to the model. Therefore, four multivariable logistic regression models were developed for each endpoint, consisting of: (1) questionnaire; (2) questionnaire and SPT; (3) questionnaire and BR testing; and (4) questionnaire, SPT and BR testing. The individual predicted probability of developing the outcome was calculated using all models. The model χ2 of the second, third and fourth models were compared to the first model’s χ2 to see whether the addition of SPT, BR testing or both of them significantly improved the questionnaire model (the level of significance was p value <0.05 using the likelihood ratio test).
The accuracy of the models was quantified using calibration and discrimination measures. Calibration, the agreement between the predicted probabilities and the observed frequencies of sensitisation, was assessed graphically and tested with the Hosmer–Lemeshow (H-L) test, where a p value of 0.10 and higher reflects good agreement.19 The discriminative ability was determined with the area under the receiver operating characteristic (ROC) curve. The ROC area can range from 0.5 (no discrimination) to 1.0 (perfect discrimination). Because the models were derived from the same subjects, the correlation between the models was taken into account when comparing the ROC area between models.20 The internal validity of the models was assessed using the bootstrapping procedure. This procedure gives a correction factor for both the model’s ROC area and for the regression coefficients of the predictors in the final model.21 The regression coefficients of the predictors in the final model were multiplied by this correction factor to prevent the model from producing optimistic predictions (that is, too high or too low estimations) when applied in future (new) workers/apprentices.
To facilitate the application of the final model for each outcome in practice, the corrected regression coefficients of the predictors in the final prognostic model were converted to easy-to-use numbers. To derive these numbers, the corrected coefficients were divided by the smallest one and rounded to the nearest half integer. The discriminative accuracy of this scoring rule was again assessed. Finally, the sums of the scores were related to their corresponding probabilities of having the endpoint. A cut-off point of the sum scores was introduced to divide the apprentices into a group with a low versus high probability of having the endpoint. All analyses were performed using SPSS 15.0 for Windows and S-Plus 6 for Windows.
There were 49 of 314 (15.6%) and 45 of 296 (15.2%) apprentices who developed new sensitisation to LA allergens and symptoms at work, respectively, at the 32nd month of the training programme. Only 16 of 279 (5.7%) apprentices developed both new sensitisation to LA allergens and symptoms at work.
Predictors of incidence of sensitisation to LA allergens
At baseline (on average at 1.4 months after entry), 140 (44.6%) students had allergic symptoms. Based on univariable analysis results, this variable was the strongest predictor of the development of new sensitisation to LA allergens (table 1). SPT-based atopy or BR were both associated with the endpoints with comparable strength.
We started the multivariable questionnaire model with four clusters of symptoms from the PCA (respiratory symptoms at work, symptoms indicative for asthma, allergic symptoms and skin symptoms). After backward selection in the regression model, only symptoms indicative of asthma and allergic symptoms appeared as independent predictors of the development of LA sensitisation and, thus, remained in the questionnaire model (table 2, model 1).
All models showed a good internal validity: from the bootstrapping procedure, we obtained correction factors of 0.91 or higher. All of them also showed a good calibration (Hosmer-Lemeshow p value >0.10). Table 2 presents the corrected regression coefficients, intercepts and ROC areas, after multiplication by the correction factor from the bootstrapping procedure. The questionnaire (model 1) had a reasonable discrimination (ROC area was 0.731; 95% CI 0.663 to 0.799). The ROC area for SPT to common allergens or BR testing as a single test was equal: 0.69 (95% CI 0.60 to 0.77). The addition of SPT-based atopy to the questionnaire model (model 2) increased the ROC area to 0.754 (delta 0.023, 95% CI −0.014 to 0.062) and significantly increased the model χ2 (deviance = 8.487, df = 1, p = 0.004). When BR testing was added to the questionnaire model (model 3), the ROC area increased by 0.046 (95% CI −0.004 to 0.098) and the model χ2 significantly increased (deviance = 16.916, df = 1, p<0.001). A complete combination of the questionnaire, SPT and BR testing (model 4) yielded a ROC area of 0.785 (delta 0.054; 95% CI −0.002 to 0.112) and significantly increased the model χ2 (deviance = 22.505, df = 2, p<0.001).
Predictors for incidence of symptoms at work
From the univariable analysis, allergic symptom was also the strongest predictor of the development of respiratory symptoms at work (table 3). Age and all symptom clusters except skin symptoms were included in the multivariable analysis. After backward selection in the regression model, only symptoms indicative of asthma and allergic symptoms remained in the final questionnaire model (table 4, model 1).
All models showed a good internal validity; correction factors from the bootstrapping procedure were 0.91 or higher. After multiplication by the correction factor, model 1 produced a good discrimination (corrected ROC area of 0.780 (95% CI 0.713 to 0.847)) and a good calibration (H-L test p value = 0.999). The ROC area of SPT to common allergens or BH testing alone was significantly lower than the questionnaire model (table 4, footnote). The addition of SPT to common allergens, BR testing, or both of them marginally increased the discriminative ability of the questionnaire model (delta ROC area ranged between 0.001 and 0.010, 95% confidence interval −0.023 to 0.043). None of the additional tests significantly increased the model χ2 of the questionnaire model (results not shown).
To evaluate the clinical relevance of the prediction models, several probability thresholds were set to classify the apprentices into low and high probability group. The accuracy of the thresholds when applied in different models for predicting LA sensitisation (table 5) and respiratory symptoms at work (table 6) was examined.
For example, if we used the questionnaire model (model 1) in the prediction of LA sensitisation, there were 84 (26.8%) apprentices with a probability of 0.30 or higher. This threshold had a positive predictive value (PPV) of 32.1% (27/84) and a negative predictive value (NPV) of 90.4% (208/230). At higher probability thresholds, the specificity and PPV of the more complex models were higher than the questionnaire model alone. Table 6 illustrates that the questionnaire model for symptoms at work showed a comparable diagnostic accuracy to that from the other models across all range of probabilities.
To apply the models in daily practice, we transformed the regression questionnaire models for both endpoints into easy-to-use score charts (see appendices 2 and 3 online). As an example of how to use this chart, a student who had symptoms indicative of asthma would have a sum score of 1 (1+0). This corresponded to a 7% probability of developing sensitisation to LA allergens and a 4% probability of developing respiratory symptoms at work. If a sum score greater than 1 is chosen as the cut-off, then this student would be included in the high probability group.
We assessed the capacity of various tools—questionnaire, skin-prick testing and bronchial responsiveness testing—alone or in combination, to correctly predict the occurrence of specific sensitisation to LA allergens and respiratory symptoms at work using data from a prospective study of apprentices.9 10 22 We found that the questionnaire models had a good accuracy in predicting incidence of LA sensitisation as well as symptoms at work. Additional information from skin-prick testing to common allergens and/or bronchial challenge to methacholine seemed to have limited predictive value in a relatively healthy population.
The questionnaire model with and without SPT and/or BR testing showed a good calibration, which meant that the predicted probabilities were in a good agreement with the observed frequency of the endpoint. It is also important to note that the questionnaire model had a comparable ability in discriminating sensitised from non-sensitised apprentices to that from SPT or BR testing alone. For symptoms at work, the discriminative ability of the questionnaire model was significantly higher than SPT as well as BR testing alone. The questionnaire model had a ROC area of 0.73, which meant that we could correctly assign a higher probability to a sensitised apprentice in 73% of pairs of apprentices in which one apprentice is sensitised to LA allergens and one is not.23 This figure was equal to what was demonstrated by the questionnaire model for sensitisation to LA allergens that was developed in a population of Dutch workers (ROC area of 0.73; 95% CI 0.65 to 0.82). In the quoted study,8 the addition of total IgE and sensitisation to common animal allergens to the final questionnaire model substantially improved the ROC area to 0.81 (95% CI 0.73 to 0.89).
We developed the models for quantification of the probability of developing sensitisation to laboratory animal allergens or respiratory symptoms at work in an animal health apprentices setting.
Information from questionnaire at entry can give an accurate prediction of the incidence of occupational sensitisation and symptoms owing to exposure to laboratory animal allergens at 32 months of the training programme. The added predictive value of skin-prick testing to common allergens and/or bronchial challenge to methacholine seemed to be limited in a relatively healthy population.
Application of the prognostic models for occurrence of occupational sensitisation and symptoms in an animal health apprentices setting may support career counselling and health monitoring programmes among individuals with a high probability.
In our study, additional information from SPT and BR testing yielded marginal to moderate additional information on the prediction of the development of occupational allergy caused by exposure to LA allergens. This might be explained by the fact that the questionnaire includes items related to both the atopic history and bronchial responsiveness. Another possible explanation was that the models were derived from a relatively healthy population. Modelling in populations with disease manifestation—for example, in the clinical setting, may yield different models with different strengths of association between predictors and the endpoint. Therefore, it might be unnecessary to confirm atopy and bronchial responsiveness with objective testing in a general population of healthy subjects; the questionnaire items representing satisfactory surrogates. However, depending upon the availability of resources, the healthcare and social system and legal context where the model will be applied, one may prefer to use a complete questionnaire-based model alone or a questionnaire in combination with additional test-based models.
As for the outcomes, we used a loose definition of respiratory symptom at work—developed any of nasoconjunctival or chest symptoms at work—due to power limitation. There were limited numbers of incident cases available for developing the models for a more complex outcome such as combination of symptoms and specific sensitisation to LA allergens. We did the PCA before modelling and after the models were developed, we did the bootstrapping procedure to check whether the models were reasonably valid. This procedure has been shown to be superior to split-sample or cross-validation methods.24 It turned out that all models had a good internal validity, the correction factors for all models were 0.91 or above; the closer the correction factor is to 1, the less optimistic (provides too high or too low prediction). Nevertheless, an external validation of the models in a new population is necessary to confirm the performance of the models and their generalisability into other groups of animal health apprentices.25 26
To the best of our knowledge, these are the first prognostic models for LA sensitisation and respiratory symptoms at work developed in an animal health apprentices setting. Such models enable the quantification of the probability of developing sensitisation to LA allergens or respiratory symptoms at work for individuals and, thus, they may support career counselling and health monitoring programmes among individuals with a high probability. At the individual level, this quantification will generate an awareness of relevant occupational allergic diseases. Apprentices with a high probability of developing sensitisation would be expected to be more vigilant—for example, use protective equipments to reduce the exposure to LA allergens. However, when applied in a working population, one should be aware of the potential negative implication of the model—for example, the predictive value is relatively low to be used in pre-employment screenings by employers or if the intervention was a change of job without signs of disease.
Another crucial issue in model application is the choice of the cut-off point above which workers will be assigned to a high probability group of developing sensitisation or respiratory symptoms at work. The choice must be based on an acceptable proportion of missed cases and of unnecessary referrals for rigorous medical tests for occupational allergies (that is, bronchial methacholine challenge test) in every worker. As shown in tables 5 and 6, a higher cut-off leads to fewer workers in the high-risk group; the specificity is higher but at the cost of lower sensitivity and vice versa. For example, when we apply the questionnaire model for sensitisation to LA allergens, we will save the expenses of unnecessary tests in almost 75% of the workers by not referring workers with a probability lower than 0.30. This cut-off point also has a high negative predictive value of 90% (208/230), which means that more than 90% of those who are not referred would indeed have a negative skin reactivity to LA allergens (if tested). However, 22 of all 49 cases will be missed. Given the nature of the disease this is not a major issue; our endpoints are not cancerous or life-threatening disease where misclassification would have dramatic impact on the prognosis. Nevertheless, if the aim is to detect as many cases as possible, a lower cut-off value might be of interest.
In conclusion, we developed prognostic models to predict the occurrence of sensitisation to LA allergens and symptoms at work in animal health apprentices. The questionnaire model alone is an easy tool that can give an accurate prediction of the incidence of occupational sensitisation and symptoms. Application of prognostic models in surveillance will enable close monitoring of the identified high-risk group for preventive purposes. Yet, the predictive capacity of the models has to be validated in other populations exposed to laboratory animal allergens before they can be used with confidence. Our models might be applicable in a new apprentice setting with exposure to other high-molecular-weight allergens, but might not be generalised to settings with exposure to low-molecular-weight agents or irritant caused by a different underlying mechanism. Therefore, it would also be interesting to validate our models in populations exposed to other high molecular weight agents, as well as to develop similar models for low-molecular-weight agents or irritant causing occupational asthma in further study.
This study was supported by the Canadian Institutes of Health Research (grant MOP-53118), the Medical Research Council of Canada (grant MT-12256) and the Institut de recherche Robert-Sauvé en santé et en sécurité du travail (grant 099-164). ES is a research fellow with a grant from the Center for Asthma in the Workplace-Canadian Institutes of Health Research.
web only appendix 66/2/111
▸ Appendices are published online only at http://oem.bmj.com/content/vol66/issue2
Competing interests: None.
Ethics approval: Informed consent was obtained from each subject and the study was performed after approval from the Sacre-Coeur Hospital’s ethics committee in accordance with Canadian ethical rules.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.