Statistics from Altmetric.com
Questionnaires are frequently used in the exposure assessment of occupational and environmental epidemiological studies. Questionnaires may be the method of choice for assessing exposure because no other sources of information are available, or because they provide the most efficient data collection method, allowing a larger study size and greater statistical power than would be possible with other more accurate measurement techniques. They may be used in combination with other methods. Information on presence of exposure (yes/no), duration, frequency, and pattern of exposure is often obtained by questionnaire. Very few, if any, standardised questionnaires that have been validated are available in this area. This is one of the major limitations of questionnaires and researchers should be extremely careful how they use self-reported exposure since it may bias the exposure-disease association. The design of new questionnaires often depends on the experience acquired with previous questionnaires.
Questionnaires require careful consideration of design and administration issues, including, for example, the length, detail of the required information, logistics, participation and completion rate, and costs involved.1,2 Exposures of interest may often be past exposures; subjects may not be able to recall these which may lead to underreporting.3 Furthermore, lack of understanding of the questions or knowledge of the exposure may bias the reporting. Recall bias, where those with disease are more likely to report their exposures compared to those without disease, even though there is no true difference, needs to be avoided; this requires careful consideration of the questions that are asked.4
ADMINISTERING THE QUESTIONNAIRE
Questionnaires can be self-administered or administered by an interviewer and either be handed or sent out, or administered over the phone or in person (face to face) respectively. This may affect the design of the questionnaire. A self-administered questionnaire is general the easiest, cheapest, and requires least involvement from both subject and researcher; it can be handed out, sent to the subjects, or computer administered. When sent out to subjects it requires a valid address of the subject; otherwise the subject may not receive it, leading to a lower response rate. The response rate and completion rate tend to be somewhat lower compared to other methods.5 Monetary incentives can increase the response rate considerably.6 The response rate can be improved by handing out the questionnaire; for example, to children in schools. Personal contact tends to increase response rates.6 Recorded delivery may improve the response rate considerably and may also provide a better indication if the subject actually lives at the address.6 Mailed out questionnaires can be easily binned or forgotten, and often require re-mailing to obtain a good response rate. The questionnaires cannot be too long since this is likely to decrease the response and completion rate.6 The subjects need to be able to complete the questionnaire within about 30–60 minutes, preferably less. The questions need to be straightforward and easy to answer, otherwise lack of understanding by the subject may result in low completion rate or inappropriate answers. The advantage of a mailed out questionnaire is that it gives subjects the time to think about the questions, and if necessary obtain further information from elsewhere. The disadvantage is that if they do not understand the question, there may not be anyone to ask unless a helpline has been set up.
Administering questionnaires in other ways, for example, by telephone interviews, has the advantage that subjects can ask the interviewer to explain any question, if necessary, but the disadvantage is that the subject may be distracted by things going on in the house or hang up when contacted. The interviewer can explain questions and explore the answers in more detail, if necessary, although perhaps to a lesser extent than in a face-to-face interview. The latter could be logistically more difficult and more expensive.
Both require a well trained interviewer, particularly to avoid interviewer bias and obtain the most relevant information. The advantage of telephone interviews compared to face-to-face interviews is that they are easier to organise and carry out as no visits are required. Nowadays most people have a phone, so the costs involved are much lower. The duration of telephone interviews should not be more than around 30–45 minutes, while face-to-face interviews can be longer than some of the other methods (1–2 hours). Both facilitate entry directly into an electronic database, which reduces the data entry effort, although it is important to avoid data entry mistakes by incorporating quality control checks. Interviews may need to be taped for quality control purposes.
Diaries (a questionnaire that is to be filled out by the subject repeatedly (for example, daily or weekly) over a period of time (for example, a few days, two weeks, or a month) to collect information on (routinely) occurring events) can be used to obtain exposure information,7,8 and they can be used to validate questionnaires.9 Diary data can be used to estimate day-to-day variability within subjects and variability between subjects,10 and hence attenuation in disease risk estimates.11 Diaries need to be short and easy to fill out. The advantage of a diary is that it needs little, if any recall, particularly if it is filled out as people go along, but they can not be used for past exposure. Diary data can be combined with measurements to obtain improved exposure estimates.12
At times it may not be possible to obtain information from the subject in the study because of death or disease, for example, Alzheimer’s disease, and it may therefore be necessary to obtain the information from a so called proxy respondent such as a relative, friend, or colleague. This may require some modification of the questionnaire, but has been carried out fairly successfully.13,14 Job title, industry, and employment periods may be reasonably accurate when reported by a proxy, but detailed information on the job, such as frequency of task or exposure to chemicals, is not. A shortened version of the questionnaire, in terms of the level of detail, may need to be developed for use by a proxy.
THE DESIGN OF QUESTIONNAIRES
The design of questionnaires needs clear aims and objectives, a selection of items that need to be translated into questions, and a logical order. Asking a question is one thing, getting a good answer another. It is extremely important to get a relevant answer of high quality to a question since otherwise time and effort are wasted. There are a number of issues that need to be considered when designing a questionnaire.1 There is often a tendency to design a questionnaire that is too long and provides so much information that not all the answers to the questions can be analysed and used in the epidemiological analysis. One of the main and most difficult questions researchers should always ask themselves is “do we really require this answer of this question for our study aims?”. If there is no good justification for it, the question should be left out. The length of the questionnaire is dependent on the way of administering the questionnaire. It is important to consider for each question how it contributes to the overall aims and objectives of the questionnaire. If it does not make a significant contribution, it should not be used in the questionnaire. Also, it needs to be clear at this stage how the answer is going to be analysed and how the answer relates to other answers in the study.2 Is useful new information obtained or does it repeat a previous question?
Researchers may add non-significant questions to questionnaires for a number of reasons, such as: (1) to evaluate recall bias by including “red herring questions”, such as a made up chemical name in a list of chemicals; (2) to investigate recall or reporting bias by including questions asking for similar information, but located in different parts of the questionnaire; and (3) to get the subject thinking in a particular direction, for example about the tasks he/she did. However, space constraints may limit these types of questions. Also it is important to note that analysis of questionnaires in many areas of epidemiological interest is often based on a single response. In contrast, the assessment of occupational exposures (in contrast to risks by job or industry title) often uses responses to multiple questions to develop a single exposure variable. This can affect how the questions are asked and any single question’s importance.
In general though, avoid questions that go into too much detail and ask for information that the subjects cannot provide, or will provide, but is not of good quality, particularly in case of past exposures. The subject may remember that they used a pesticide a number of years ago and the broad category, but they are unlikely to remember the name of the pesticide or the active ingredient.15 Other questions that are difficult to answer are questions that require calculations (for example, hours/year) and questions on rare and relatively insignificant events. In certain situations it may be helpful to use more than one questionnaire (for example, a generic questionnaire) and additional questionnaires (for example, specific job questionnaires).16 The latter can be used for more in-depth questions on particular topics after the subject indicates in the former that these take place. This kind of approach has been used for occupational questions in community based studies. The researcher should make the decision prior to developing a questionnaire as to whether a small number of substances or many substances will be investigated and what the desired exposure outcome variable is (for example, presence/absence of a substance, or quantitative estimates).
The longer the recall time, the less detail can be expected from the subject and the less likely it is that the subject will give an answer of sufficient quality to be included in the analysis. Complex exposure scenarios may occur, for example, where the subject is exposed to many different substances over different time periods. This makes it difficult to design a simple and straightforward questionnaire and the complexity may overwhelm the subject and reduce the quality of the answers. In this case it is important to focus on key events. Rare events may only be remembered by the subjects if they made a large impact on their life or are somehow connected to other events.
As mentioned, there are few, if any standardised questionnaires for exposure assessment that have been validated. Many researchers therefore start from scratch, or base their design on their previous experience with questionnaire data. A critical evaluation of questionnaires that have been used in previous studies in the particular area, including the response rate, how the answers were used in the analyses, and the potential for misclassification will be very helpful in the design of a new questionnaire. At times it is advisable to use the same questionnaire as in a previous study, even though it could be improved, so that an exact comparison can be made with results of a previous study. The design of questionnaire is often a very long process and many changes will be made along the way.
OPEN ENDED OR CLOSED ENDED QUESTIONS
Open ended questions are questions without restrictions on the answers of the subjects, while closed ended questions have restrictions in the form of a limited number of possible answers; for example, categories that subjects have to pick (figs 1 and 2). Open ended questions are generally used to record simple factual information such as name, weight, age, and occupational title. Sufficient amount of space should be left to answer the question. The answer can be considerably different depending on whether the question is a closed or open ended question.17
For closed ended questions the answer may differ depending on the number of categories that are provided as a possible answer. For example, if the question is asked “How often did you wash your hands?” and responses of <10/wk, 10–30/wk, >30/wk are provided, the distribution of subjects may dominate the middle category and there may be few subjects in the two extreme categories. In addition, categories can “lead” a subject to a response. For example, if you are considering asking for the number of times cleaning the workplace floor in a week as either 1/wk, 2/wk, 3 or 4/wk, and 5/wk or more, or <1–2/wk, 3–5/wk, 6–10/wk, and 11/wk or more, the second categorisation could result in a different distribution of subjects than the first. Subjects tend not to want to be on the extreme end, and so they might respond 3 or 4/wk to the first categorisation, but with the second categorisation they might respond 3–5 or 6–10/wk. The categorisation may also not reflect the subject’s experience since he/she cleans many times a day, and therefore thinks that weeks should be read as days. The effect of the category definition can be reduced by knowledge of the expected frequencies of the responses. However, most of the evidence on categorisation comes from outside the field of occupational and environmental epidemiology and little research has been done specifically within this field.
For closed questions make sure to avoid potential overlap;18 the answers need to be mutually exclusive, although the respondent may be able to pick more than one answer. Open ended questions generally need to be coded by a trained researcher before they can be analysed, for example, occupation or tasks they did; this may take a lot of time and careful interpretation of some of the answers and may itself introduce bias.
The words used in the questionnaire should be understood by the subjects and they should neither to be too difficult nor too simple and meet the reading level of the target audience.1 The questions should be clear, unambiguous, short, and to the point, and the subject should not have to figure out what is actually being asked.18 There should be no unexplained or vague terms, jargon, or abbreviations, although at times jargon may be needed for specialised jobs or tasks. Wording such as rarely, sometimes, regularly, almost always, and always should be avoided, since it may be interpreted differently by different subjects, and more exact and objective wording should be used. Only one question at a time should be asked and the questions should be unbiased. Questions cannot be too precise and the subject needs to be familiar with them. The answers should be able to be linked to the aims and objectives of the questionnaire.
EXAMPLES OF QUESTIONS THAT HAVE PROBLEMS
“Do you work with solvents regularly?”. This appears to be a straightforward question but it is unclear what is meant by “regularly” or by “work”. In this case it would be better to define what is meant by “regularly”, for example, at least more than one hour once a day or once a week. Does “work” mean “handle” solvents or work in environment where others work with them? Also, what is meant by solvents? All solvents, or only organic solvents?
“Where does your highest exposure to NO2 take place”. Many subjects will not know what NO2 is and even writing it down as “nitrogen dioxide” will not help. Furthermore, in general people will have little knowledge where nitrogen dioxide occurs, let alone where their highest exposure takes place. In this case surrogate questions should be used; for example, questions on the use of gas cookers or living within 100 metres of a main road, which are both associated with nitrogen dioxide exposure.
“How many minutes per week are you exposed to traffic fumes?”. This question is likely to be too precise and they will probably have to estimate first their duration each day and then add them up to get a total for each week, which may be too complicated in many cases. Also subjects are unlikely to know exactly what they should count as traffic fumes. Are they exposed to traffic fumes when they are on the job, such as for garage mechanics, office workers of a car dealership, or parking garage attendants, or when walking along a road, when they sit inside a car, when they live near a main road, or in all these cases?
“Do you mix, load, and apply pesticides, and how often?”. Here we have a number of questions in one question and these should be separated. For example:
(1a) Do you mix and load pesticides?
Yes (go to question 1b)
No (go to question to 2a)
(1b) How often do you mix and load pesticides per week?
(2a) Do you apply pesticides?
Yes (go to question 2b)
No (go to question 3)
(2b) How often do you apply pesticides per week?
Mixing/loading is a task of pesticides applicators and generally has a different level of exposure than application and should be separated. Mixing and loading are also two different tasks, but are generally combined because of the short duration of each. It is also tempting to ask how long the subject mixes and loads pesticides, but this is difficult for a person to know without timing this. Without timing the answer often will be 5 or 10 minutes but it is questionable if this is the true duration, and it may not be very useful in any analyses since it is very similar for everyone. On the other hand it is unlikely to be far off the actual duration. The frequency of mixing and loading is likely to the more important factor when estimating the total duration of mixing and loading per week, and generally adding a question on the duration, which is likely to result in an inaccurate answer, may be a waste of space. The application is generally of longer duration and this can be asked. Another source of possible misinterpretation on a “how often” question is do you mean per day or total? For example, how do I count the frequency if I mix/load three times in one day for the same field versus mix and load once for my lettuce field, mix once for my green beans field, and load once for my carrot field; what if I always mix the pesticide, but sometimes someone else loads it? In addition, it may not be clear whether the question applies to the subject personally or, for example, to the subject’s family. Also, do you want information if the subject did not apply pesticides, but worked in an area where pesticides were applied?
“Have you used any pesticides?”. First, it is unclear in this question to which time period it applies. Second, many subjects, particularly in community based studies will not know what a pesticide is exactly and what they should count. Besides home and garden pesticides do they include herbicides, shampoos against lice, pet collars against fleas, fly spray, mothballs, or bleach? Do they need to include both organic and non-organic pesticides? This apparently simple question is unlikely to provide a satisfactory and easy interpretable answer and should therefore be explained in more detail and broken up into a number of questions.
“Did you use any pesticides in 1973?”. Besides the problems discussed above, the additional problem in this case is the very precise time period, which is a long time ago. People are unlikely to remember this kind of detail many years ago. Introduction questions to aid their memory may help at times; for example, if interested in garden pesticides the researcher could ask if they had a garden in that particular year, followed by questions on particular pests they may have had.
FORMAT OF THE QUESTIONNAIRE
The format of the questionnaire is as important as the actual questions. Remember that it is easy for people to bin the questionnaire. In the case of paper questionnaires, long and complex looking questionnaires do not encourage the subject to start filling them out. Make sure that that the questionnaire looks nice, tidy, and appealing, but do not go over the top, and use large enough letter type and an easy to read font.1 Using different colours for different questionnaires can be helpful at times. Make sure to provide clear instructions on how to answer the questions, but do not make them too long since this will put off the subject. Use a logical order for your questions and start with some simple questions first to get the subject going. Use the order in such a way that the subjects keep an interest and that there is a natural flow. The former may be difficult at times when asking many exposure related questions, particularly when a subject does not know how this relates to a particular disease or, in the case of a case-control study, when asking controls. The response rate among controls often tends to be lower compared the response rate among cases. Subjects are often not interested or do not know about chemicals, for example, but they know about particular events (for example, job, moving house, births) in their life and this could be used in the questionnaire to obtain relevant information.19 Not only may these events be related to particular exposures, but they may also keep the subject interested and increase response rates.6 Thank the subjects at the end and remind them what to do with the questionnaire.
Branching off (or screening)—that is, going into more detail for particular questions is not easy with self-administered paper questionnaires since it will take up a lot of space on a page and a subject may get confused about what to answer. Arrows or colour coding may help. However, it is possible to do this with computers. This could be very useful and provide more specific information; for example, on occupational exposures.16
The use of matrices can be very helpful at times; for example, when asking for a job history—that is, for rows for each job, and columns for job title, industry, tasks, and start and finishing date. A job title and type of industry can provide substantial information about potential exposures and can be coded into standard coding systems (for example, Standard Occupational Classification 200020). However, job titles used may vary between companies; for example, they may include different tasks. If possible, specific information, such as the frequency and use of protective equipment may need to be obtained about particular tasks that the subject carried out, particularly those that may be related to the exposure of interest.16 Job history matrices can be coded and expressed as a job exposure matrix (fig 3), where the researcher codes the job information and relates it to certain exposures.21 Trained experts are required in this case to set up a job exposure matrix and carry out the coding, which may be a laborious task and provide mixed results.21
For a good coding of occupation it is generally necessary to have a few open ended questions on company, job, and activities. For open ended questions in the questionnaire it is important to make a coding scheme early on in the study, which helps to focus the analyses, but it may need to be modified in the light of unexpected responses. Humans make mistakes and therefore a good quality control scheme needs to be carried out.
AIDS TO RECALL
Multiple choice questions or cards with alternative answers given by the researcher may be very helpful, particularly when there are a limited number of possible answers. A tiered approach can be very helpful; for example, when trying to estimate the exposure to a particular chemical at work, the questionnaire first may ask:
The employer the subject worked for
Type of industry
The particular task that he/she carried out
The type of chemicals that he/she worked with (e.g. pesticides, metals, solvents)
The names of chemicals that he/she worked with.
This approach has a further advantage that even if the subject does not know the answer to the last question, answers from the previous questions could be used for some analyses in an epidemiological study; for example, disease in relation to type of industry or type of industry and job title, and also to predict what kind of substance the respondent worked with.
A calendar may also be used as an effective aid to recall. When studying risk factors of birth outcomes, a calendar could be helpful to determine where subjects lived or where they worked before conception and during different stages of various pregnancies. Pregnancies and everything around them are events that are generally clearly remembered.22
The design of questionnaires needs clear aims and objectives, a selection of items that need to be translated into questions, and a logical order.
The words used in the questionnaire should be understood by the subjects, they should neither be too difficult nor too simple, and they should be clear, unambiguous, and concise.
The format of the questionnaire is as important as the actual questions.
Pilot testing is an extremely important part of questionnaire design, and a sufficient amount of time should be allocated to it.
The validity and repeatability of the questionnaire should be assessed, where possible.
Pilot testing is an extremely important part of questionnaire design, and a sufficient amount of time should be allocated to it.23 Before pilot testing the questionnaire on people representative of the target population, it should be evaluated by a number of other researchers, particularly those who have used similar questionnaires and used the answers in an epidemiological study. This may be followed by a sample of convenience; for example, relatives, friends, or colleagues. After this, the questionnaire should be tested on a sample of people representative of the target population. At this stage it may become clear if the intentions of the researchers are sufficiently understood by the subjects. In some cases it may take a number of pilot tests to get the questionnaire right, and the researchers should be critical of their work and open to suggestions at this stage. The better the pilot testing the less regrets there will be at the end of the study. Throughout the process written comments should be obtained that can be evaluated by a number of researchers.
In the pilot testing issues that are evaluated are the flow of the questions, whether the words are understood, and whether the questions are interpreted similarly. This at times may be difficult to establish and it may need some more in-depth discussion with the pilot subjects to determine how they interpreted the question and what they thought when giving an answer. Cognitive interviewing may help in this process and improve the questionnaire.18 Also, the researchers need to evaluate how the answers can be interpreted and if they can be analysed and used for the epidemiological study. This is extremely important at this stage, particularly since questions can still be changed. More simple facts, such as if all the questions were answered, are easier to establish.1
More and more, international multicentre studies are carried out, for example in Europe, and this may require translation of questionnaires into different languages. Furthermore, a (large) influx of immigrants in the country being studied often also requires translation of the questionnaire into other languages since they may not be sufficiently familiar yet with their new language to be able to answer any questions satisfactorily. In these cases, the questionnaire should be translated and back translated, preferably by a number of experts familiar with the language and the topic. The researchers should be aware of cultural differences and take this into account in the questionnaire. After the translation has been carried out, the questionnaire should be pilot tested again to make sure that no information has been lost in the process and the questions are interpreted in the same way as in the original questionnaire.
A questionnaire may look good and the response to the questions is good, but does the questionnaire actual measure what it needs to measure—that is, what is the validity of the questionnaire? For example, are those reporting longer or higher exposure to environmental tobacco smoke (ETS) actually exposed to more ETS; are those reporting working with solvents actually exposed to (higher) solvent levels; do people that report drinking five glasses of tap water per day actually drink five glasses of tap water per day? Ideally all questions and answers should be validated—that is, compared to a gold standard. However, often this is not possible; for example, because the questions relate to exposure in the past, there is no method to measure the substance of interest, or it is unclear what the gold standard is. Selecting a valid gold standard is not necessarily easy and if a poor one is selected, it can provide misleading information about the questionnaire. Issues of timing and variability of the gold standard can affect the evaluation.
Up to a certain extent, diaries can be used to validate a water ingestion questionnaire,9,10,24 personal exposure monitoring of nicotine to validate ETS exposure,25,26 or biomonitoring of solvent metabolites (hippuric acid or methylhippuric acid) in urine to validate aromatic solvent exposure.27 The validation may only need to be carried out on a proportion of the population, and may provide invaluable information for the interpretation of the epidemiological study. However, “gold standards” may only be relevant to current exposures and depend on the half-life in the body of the substance being measured and specificity of the biomarker. The sensitivity, specificity, and predictive value of the various questions or the correlation between different questions from different methods can be determined,17 and therefore the extent of exposure misclassification, if any, and effect on the risk estimates.11 Unfortunately there appears to be only a limited amount of information in this area and more work is needed.
Tielemans and colleagues27 used biomonitoring to assess the validity of their various questionnaire methods and found sensitivity and specificity coefficients of around 0.30–0.55 and 0.77–0.92 respectively when using a generic questionnaire for solvent exposure, while this increased to 0.40–0.70 and 0.75–0.93 respectively when using a detailed job specific questionnaire, which elicited details on occupational tasks, products, and frequency of activity. The highest positive predictive value was found for the job specific questionnaire (0.52). Shimokura and colleagues10 compared questionnaire data and diary data for water related activities and found that there was a good correlation between the drinking water intake (r = 0.78) and for time spent showering (r = 0.68), but found that the actual amount of reported drinking water intake was considerably higher when using the questionnaire compared to the diary (0.75 v 0.40 l/day). The difference for showering was less (10.5 v 9.8 min/day). Eisner and colleagues25 found a moderate correlation (r = 0.47) between questionnaire reporting of ETS and personal measurements of nicotine, while Coghlin and colleagues26 found a much higher correlation (r = 0.91). A possible explanation of the difference may be the higher nicotine levels in the former study compared to the latter, which shows the importance of taking into account population characteristics. Teschke and colleagues17 found differences in the performance in questionnaires depending if they were closed or open ended questions.17 The former generally showed higher sensitivity, although slightly lower specificity.
Besides the validity, there is the repeatability of the questionnaire—that is, are the answers reproducible, do we always get the same answer when we administer the questionnaire repeatedly? (at times the term reliability or reproducibility is used here too). This can be assessed by providing the questionnaire twice to a proportion of subjects in the target population, and estimate, for example, the correlation between the answers.7,24,28–30 The question that is often raised is whether the reproducibility of the questionnaire is measured or variable in exposure (this is not an issue if the questionnaire administered the second time addresses the same time frame as the questionnaire administered the first time). The interval between the two occasions should be long enough to provide independent observations, but not too long to avoid true variation in exposure. Some researchers tried to assess the reproducibility of a questionnaire, but used different methods of administration; for example, mailing and telephone or self-administered questionnaire and a face-to-face interview. This may introduce additional variation into the process and should be avoided. Künzli and colleagues7 used a number of different questionnaires to assess long term ambient ozone exposure and found some differences in the reproducibility, showing the importance of assessing different questionnaire methods. Barbone and colleagues24 found a good reproducibility for their questionnaire on tap water related activities (r = 0.6–0.9).
Questionnaires are an important instrument in the exposure assessment of occupational and environmental epidemiological studies. The development of questionnaires needs to address many issues and is generally a long process. This paper can only give a flavour of what is involved and further reading is suggested. Certainly, further work is needed on the validation of questionnaires.
QUESTIONS (SEE ANSWERS ON P 214)
Questions in the questionnaires should be:
Response rate for questionnaires can be increased by:
Increasing the size of the questionnaire
Validity of the questionnaire can be measured by administering the same questionnaire repeatedly:
Proxy respondents may be needed in case of death from diseases such as Alzheimer’s:
In case of pesticides, those using pesticides may remember:
That they used a pesticide a number of years ago
The broad category of pesticides
The name of the pesticide
The active ingredient
For a job exposure matrix one needs the following information:
Trained expert to code
There are many well validated standardised questionnaires for exposure assessment in occupational and environment epidemiology:
A large part of this paper is based on the chapter on “Questionnaires” in Exposure assessment in occupational and environmental epidemiology (Nieuwenhuijsen MJ, ed. Oxford: Oxford University Press, 2003) and a number of chapters from Principles of exposure measurement in epidemiology (Armstrong BK, White E, Saracci, R. Oxford: Oxford University Press, 1992)
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.