Effect of the Method of Administration, Mail or Telephone, on the Validity and Reliability of a Respiratory Health Questionnaire. The Spanish Centers of the European Asthma Study

doi:10.1016/S0895-4356(98)00063-8

Journal of Clinical Epidemiology

Volume 51, Issue 10, October 1998, Pages 875-881

https://doi.org/10.1016/S0895-4356(98)00063-8 Get rights and content

Abstract

The European Community Respiratory Health Survey (ECRHS), a multinational survey, assesses and compares the prevalence of asthma among subjects, aged 20 to 44, in several European areas. In Spain, some participating centers have used mail and telephone as methods of questionnaire administration. The objective of the present study was to determine whether the validity and reliability of the questionnaire differed by method of administration. Reliability of the questionnaire was measured with the kappa index and the odds ratio of agreement, and validity with the sensitivity and specificity. This study found differences in the reliability of the questionnaires although these differences were more related to the questions themselves than to the method of administration. Among men, but not women, mailed questionnaires were more sensitive and telephone questionnaires more specific. We hypothesize that these differences in validity were due to the self-selection to more severe symptomatic subjects replying earlier and therefore to the mailed questionnaire. Combining different methods of administration was useful as it increased participation and was an adequate procedure to obtain information of good quality.

Introduction

Personal interview is a commonly used method of questionnaire administration and considerable research has been done on the interview process [1]. Compared to other methods, the major disadvantage of the interview is its expense 2, 3, 4. Mail, telephone, and video questionnaires have reduced costs, but their use has been challenged because they achieve lower questionnaire accuracy and lower response rate than interviews 3, 5, 6. Another strategy is to use a combination of different methods of administration 7, 8, 9, which results in a wider population coverage, lower nonresponse rate, and lower costs. One shortcoming of this last approach is that a new source of error may be introduced if the quality of the information obtained from the questionnaire differs with the method of administration. When different methods of administration have been compared, both differences 3, 10, 11 and no differences 2, 12 have been reported, which also occurred with the respiratory questionnaires 13, 14, 15, 16.

The ECRHS (European Community Respiratory Health Survey) is a multinational survey designed to assess and compare the prevalence of asthma in subjects, aged 20 to 44 years, in several European areas. While centers from northern and central Europe achieved high response rates administering the questionnaire by mail, others in Spain, Italy, and Portugal required the additional use of telephone [17]. The objective of the present study was to assess whether the validity and reliability of the questionnaire used in the Spanish centers of the ECRHS differed by method of administration (mail and telephone).

Section snippets

Methods

The ECRHS is a multinational survey set up to compare the prevalence of asthma in subjects, aged 20 to 44 years, in different European areas. The design and recruitment procedures used in the full study have been described elsewhere [18]. In brief, the ECRHS is a two-phase population-based survey that in the first phase administered a short questionnaire about respiratory symptoms to a random sample. Subjects reporting at least one of three specific symptoms in this questionnaire were

Results

The descriptive characteristics of the samples used in the reliability and in the validity study are presented in Table 1. In the reliability study, the mean age did not differ by method of administration (31.9 by mail and 31.6 by telephone). A higher proportion of the subjects classified as “symptomatic” responded by mail (57.9% by mail versus 42.1% by telephone). A higher proportion of questionnaires was obtained by mail in Albacete and Barcelona, while in Galdakao the majority was obtained

Discussion

This study found differences in the reliability of the short questionnaire although these differences were more related to the questions themselves than to the method of administration. When validity of the questionnaire was assessed, mailed questionnaires were more sensitive and telephone questionnaires were more specific for males, but not for females.

Most of the questions achieved a good agreement (kappa around 50%) in both the mailed and telephone administration, although only one question

Acknowledgements

The authors thank Patricia Stephens for editorial evaluation and Dr. Jonathan Samet for his review and insightful comments, on an earlier version of this manuscript.

The Spanish Group of the European Asthma Study centers and investigators: J. M. Antó (Principal Investigator); Albacete: J. Martinez-Moratalla, E. Almar, M. Arevalo, A. Mateos, A. Sanchez, M. Vizcaya; Barcelona: J. Sunyer, F. Burgos, J. Castellsagué, B. Galobardes, J. Roca, J. B. Soriano, A. Tobias; Galdakao: N. Muniozguren, M.

References (33)

J.K. Peat et al.
Reliability of a respiratory history questionnaire and effect of mode of administration on classification of asthma in children
Chest
(1992)
S.S. Coughlin et al.
The logistic modeling of interobserver agreement
J Clin Epidemiol
(1992)
S. Macintyre
Gender differences in the perceptions of common cold symptoms
Soc Sci Med
(1993)
F.M. Bush et al.
Analysis of gender effects of pain perception and symptom presentation in temporomandibular pain
Pain
(1993)
C.F. Canell et al.
A Summary of Studies of Interviewing Methodology
(1977)
J.R. Hochstim
A critical comparison of three strategies of collecting data from households
J Am Statis Assoc
(1967)
M.F. Weeks et al.
Personal versus telephone surveys for collecting household data at the local level
Am J Public Health
(1983)
B.I. O’Toole et al.
A comparison of costs and data quality of three health survey methodsMail, telephone and personal home interview
Am J Epidemiol
(1986)
R.A. Shaw et al.
Comparison of a video questionnaire with the IUATLD written questionnaire for measuring asthma prevalence
Clin Exp Allergy
(1992)
R.M. Mickey et al.
Comparability of telephone and household breast cancer screening surveys with differing response rates
Epidemiology
(1994)

D.J. Brambilla et al.

A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey

Am J Epidemiol

(1987)

J. Siemiatycki et al.

Quality of response in different population groups in mail and telephone surveys

Am J Epidemiol

(1984)

C.A. McHorney et al.

Comparisons of the costs and quality of norms for the SF-36 health survey collected by mail versus telephone interviewResults from a national survey

Med Care

(1994)

A. Cartwright

Who responds to postal questionnaires?

J Epidemiol Community Health

(1986)

C.S. Aneshensel et al.

Telephone versus in-person surveys of community health status

Am J Public Health

(1982)

L.W. Chambers et al.

Sensitivity to change and the effect of mode of administration on health status measurement

Med Care

(1987)

Cited by (42)

Patient-physician communication about work-related asthma: What we do and do not know
2015, Annals of Allergy, Asthma and Immunology
Citation Excerpt :
Participation in the Asthma Call-Back Survey could be subject to selection bias because BRFSS participants with asthma were called back for the Asthma Call-Back Survey and respondents with more severe disease might self-select to participate. The authors hypothesize that these respondents might be more active in discussing asthma in relation to work with their physicians, thus overestimating the present results.50 Also, information on asthma and WRA was self-reported and not validated by medical records or follow-up with health care providers; thus, estimates could be subject to misclassification.
Effective patient–physician communication is the key component of the patient–physician relationship.
To assess the proportion of ever-employed adults with current asthma who talked about asthma associated with work with their physician or other health professional and to identify factors associated with this communication.
The 2006 to 2010 Behavioral Risk Factor Surveillance System Asthma Call-Back Survey data from 40 states and the District of Columbia for ever-employed adults (≥18 years old) with current asthma (N = 50,433) were examined. Multivariable logistic regression analyses were conducted to identify factors associated with communication with a health professional about asthma and work.
Among ever-employed adults with current asthma, 9.1% were ever told by a physician that their asthma was related to any job they ever had and 11.7% ever told a physician or other health professional that this was the case. When responses to the 2 questions were combined, the proportion of those who communicated with a health professional about asthma and work was 14.7%. Communication with a health professional about asthma and work was associated with age, race or ethnicity, employment, education, income, insurance, and urgent treatment for worsening asthma.
A small proportion of patients with asthma might communicate with a health professional about asthma associated with work. Future studies should examine whether patients with asthma ever discussed with a health professional the possibility that their asthma might be related to work to provide information on the frequency of patient–clinician communication about asthma related to work.
Asthmatics and ex-smokers respond early, heavy smokers respond late to mailed surveys in Italy
2010, Respiratory Medicine
Citation Excerpt :
The promptness to respond, observed in the present study among ex-smokers, had been already reported in the Seventies by an American study,3 where the prevalence of ex-smokers decreased from 19.6% in early responders to 12.1% in late responders. However this finding was not confirmed by subsequent studies.6,20 Anyway, the observation that ex-smokers are more willingly to participate in health surveys is not surprising as ex-smokers seem to be more aware of health problems21: indeed, it is known from the current literature that becoming an ex-smoker without smoking-related disturbances involves a great improvement in health education, such as shedding self-exempting beliefs typical of smokers and accepting information about the diseases caused by smoking.22
Response to mailed epidemiological surveys has decreased in recent decades. Since subjects with respiratory symptoms are usually early responders to surveys performed in Southern Europe, this trend could bias prevalence estimates. The present study aimed at evaluating the impact of non-response bias on prevalence estimates of respiratory symptoms and smoking habits.
In 9 centres, participating in the Italian Study on Asthma in Young Adults (ISAYA), random samples of people aged 20–45 years were administered a mailed questionnaire between 1998 and 2000. Non-responders were contacted again first by mail and then by phone. Cumulative response percentage was 30.5%, 52.4% and 72.7% (18,873/25,969), respectively, after the 1st, 2nd and 3rd contact.
The prevalence of self-reported current asthma, asthma-like symptoms, and chronic cough/phlegm was more than halved from the first contact (5.6%, 17.8%, 14.6% respectively) to the third contact (2.7%, 6.4%, 6.9%). This pattern was less pronounced when considering allergic rhinitis and past asthma, whose prevalence decreased, respectively, from 21.5% to 15.6% and from 3.5% to 2.6%. At the same time the proportion of current smokers increased from 29.2% to 38%, while the proportion of ex-smokers decreased from 16.5% to 10.1%. In a multinomial logistic model current asthma, asthma-like symptoms, chronic cough/phlegm and smoking habits, and to a lower extent past asthma and allergic rhinitis, were significant predictors of late response.
In Italy when response percentage is low, the prevalence of current asthma, chronic cough/phlegm and ex-smokers is overestimated, while the proportion of current smokers is underestimated.
Electronic mail was not better than postal mail for surveying residents and faculty
2005, Journal of Clinical Epidemiology
To compare response rate, time to response, and data quality of electronic and postal surveys in the setting of postgraduate medical education.
A randomized controlled trial in a university-based internal medicine residency program. We randomized 119 residents and 83 faculty to an electronic versus a postal survey with up to two reminders and measured response rate, time to response, and data quality.
For residents, the e-survey resulted in a lower response rate than the postal survey (63.3% versus 79.7%; difference −16.3%, 95% confidence interval (95% CI) −32.3% to −0.4%%; P = .049), but a shorter mean response time, by 3.8 days (95% CI 0.2–7.4; P = .042). For faculty, the e-survey did not result in a significantly lower response rate than the postal survey (85.4% vs. 81.0%; difference 4.4%, 95% CI −11.7 to 20.5%; P = .591), but resulted in a shorter average response time, by 8.4 days (95% CI 4.4 to 12.4; P < 0.001). There were no differences in the quality of data or responses to the survey between the two methods.
E-surveys were not superior to postal surveys in terms of response rate, but resulted in shorter time to response and equivalent data quality.
Mail surveys resulted in more reports of substance use than telephone surveys
2005, Journal of Clinical Epidemiology
Citation Excerpt :
Mail surveys are far less expensive, but how they compare with telephone surveys in terms of collecting sensitive substance-use information is unclear. Several studies have compared the mail and telephone survey responses on personal issues such as asthma [4], sexual behavior among gay males [5], prostate surgery [6], functional limitations [7,8], urinary symptoms [9], and anxiety [10]. In general, these studies find that telephone respondents give more socially desirable responses, and mail respondents report more problems or socially proscribed behaviors.
To determine to what extent the substance-use information obtained in surveys is affected by method of data collection.
Questions on the use of alcohol and drugs were administered to samples of Minnesota adults assigned to one of two conditions to test the effect of mode of administration (mail and telephone); 816 persons completed the survey, roughly one half by mail and one half by telephone.
Those interviewed by telephone revealed more heavy use of alcohol, but the mail sample includes disproportionate numbers of respondents from demographic groups that exhibit less use. Controlling for these differences across modes, as well as the differential use of listed telephone numbers and addresses, reduces the effect of mode on one measure of heavy alcohol use to nonsignificance but yields significant effects of mode on others. Specifically, those in the mail condition reported higher levels of illicit drug use in the last year, alcohol use in the last month, and heavy alcohol use in the last 2 weeks.
The greater, and arguably more accurate, reporting of substance use, coupled with potential cost savings, suggests that researchers should consider using mail surveys for investigating substance use.
Validation of the Arabic Version of the European Community Respiratory Health Survey Screening Questionnaire
2023, Journal of Asthma and Allergy
Models for estimating and projecting global, regional and national prevalence and disease burden of asthma: Protocol for a systematic review
2017, BMJ Open

View all citing articles on Scopus

^☆: Accepted for publication on 28 April 1998.

View full text

Original ArticlesEffect of the Method of Administration, Mail or Telephone, on the Validity and Reliability of a Respiratory Health Questionnaire. The Spanish Centers of the European Asthma Study☆

Abstract

Introduction

Section snippets

Methods

Results

Discussion

Acknowledgements

Chest

J Clin Epidemiol

Soc Sci Med

Pain

A Summary of Studies of Interviewing Methodology

A critical comparison of three strategies of collecting data from households

J Am Statis Assoc

Personal versus telephone surveys for collecting household data at the local level

Am J Public Health

A comparison of costs and data quality of three health survey methodsMail, telephone and personal home interview

Am J Epidemiol

Comparison of a video questionnaire with the IUATLD written questionnaire for measuring asthma prevalence

Clin Exp Allergy

Comparability of telephone and household breast cancer screening surveys with differing response rates

Epidemiology

A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey

Am J Epidemiol

Quality of response in different population groups in mail and telephone surveys

Am J Epidemiol

Comparisons of the costs and quality of norms for the SF-36 health survey collected by mail versus telephone interviewResults from a national survey

Med Care

Who responds to postal questionnaires?

J Epidemiol Community Health

Telephone versus in-person surveys of community health status

Am J Public Health

Sensitivity to change and the effect of mode of administration on health status measurement

Med Care

Original Articles
Effect of the Method of Administration, Mail or Telephone, on the Validity and Reliability of a Respiratory Health Questionnaire. The Spanish Centers of the European Asthma Study☆