You are here

Article Text

Article menu
XML
Short report
Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions
  1. Stefano Mattioli1,
  2. Andrea Farioli1,
  3. Robin M T Cooke1,
  4. Alberto Baldasseroni2,
  5. Jani Ruotsalainen3,
  6. Donatella Placidi4,
  7. Stefania Curti1,
  8. Gianpiero Mancini5,
  9. Mauro Fierro1,
  10. Giuseppe Campo6,
  11. Francesca Zanardi1,
  12. Francesco S Violante1
  1. 1Section of Occupational Medicine, Department of Internal Medicine, Geriatrics and Nephrology, University of Bologna, Italy
  2. 2Tuscany Regional Centre for Occupational Injuries and Diseases (CeRIMP), Florence, Italy
  3. 3Finnish Institute of Occupational Health, Cochrane Occupational Safety and Health Review Group, Kuopio, Finland
  4. 4Department of Experimental and Applied Medicine, Section of Occupational Medicine, University of Brescia, Brescia, Italy
  5. 5Department of Prevention, Occupational Health Unit, Azienda USL di Ravenna, Ravenna Local Health Authority, Ravenna, Italy
  6. 6INAIL (Italian Workers' Compensation Authority), Area Ricerca, Dipartimento Processi Organizzativi, Rome, Italy
  1. Correspondence to Dr Francesca Zanardi, Alma Mater Studiorum, University of Bologna, Unità Operativa di Medicina del Lavoro, Policlinico S.Orsola-Malpighi, Via Pelagio Palagi 9, 40138 Bologna, Italy; francesca.zanardi3{at}unibo.it

Abstract

Objective To compare the yield of hand-searching with optimised electronic search strategies in retrieving occupational health (OH) intervention studies published in a language other than English.

Methods The authors systematically hand-searched and screened reports of OH intervention studies published in Italian in peer-reviewed scientific journals between 1990 and 2008. The authors evaluated how many of them met the Cochrane Occupational Safety and Health Review Groups (OSHRG) definition of being an OH intervention study and how many potentially relevant studies retrieved by hand-searching would not be found by PubMed alone using the OSHRG's most specific and most sensitive search strings.

Results Hand-searching retrieved 25 articles (reporting 27 studies), including nine not indexed in MEDLINE. Most studies (81%, 22/27) had a before–after design and only one was a randomised trial. The OSHRG's most sensitive search string retrieved all 16 articles published in the Italian language journals that were indexed in MEDLINE, while the most specific search strategy retrieved nine articles (56%, 9/16). The most specific search string showed a lower ‘number needed to read’ value than the most sensitive one (60 vs 132).

Conclusions These findings suggest that a sensitive electronic search strategy may be able to find most of the OH interventions published in languages other than English that are indexed in MEDLINE. Hand-searching of important national journals not indexed in MEDLINE should be considered when conducting particularly in-depth research.

  • Back disorders
  • statistics
  • ergonomics
  • epidemiology
  • preventive medicine
  • occupational health practice
  • statistics
  • exposure assessment
  • asbestos
  • accidents
  • psychology
  • health and safety
  • OH services
  • intervention studies
  • healthcare workers
  • cardiovascular
  • bronchitis
  • tuberculosis
  • cancer
  • international occupational health

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/3.0/ and http://creativecommons.org/licenses/by-nc/3.0/legalcode

https://doi.org/10.1136/oemed-2011-100180

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    What this paper adds

    • The evaluation of the effectiveness of occupational health (OH) interventions has become a fundamental step towards evidence-based OH practice.

    • The number of intervention studies available in the field of OH is relatively modest.

    • For journals indexed in MEDLINE, searching PubMed with a sensitive and optimised search strategy will locate most of the OH intervention studies, even if published in a language other than English.

    • When conducting particularly in-depth research (eg, a systematic review on a little studied intervention), hand-searching of important national journals not indexed in MEDLINE should be considered to maximise coverage of the search.

    Introduction

    As the culture of evidence-based medicine has led to widespread support for an evidence-based approach to occupational health (OH), the evaluation of the effectiveness of OH interventions has become a fundamental step towards evidence-based OH practice.1 ,2 One key aspect of evidence-based practice is the ability to conduct effective and efficient literature searches. In the OH field, this challenge has been addressed in at least two ways: with the development of systematic search strategies to optimise the yield of electronic databases, 3–5 and by trying to identify a set of core OH journals.6 ,7 Both approaches can help occupational physicians and other stakeholders in OH to trace results published in the English language on which interventions work and which do not. However, some peculiar aspects of OH limit the completeness of the results provided by literature searches based solely on a restricted set of core journals or on electronic resources. Many OH interventions are implemented by local health authorities or administrations rather than by research centres such as universities, national institutes or private enterprises. In such cases, the appropriateness of the research methodology and scientific reporting may lag behind the quality of the interventions themselves. Therefore, evaluations of well-designed and conducted interventions are sometimes reported in secondary scientific journals not covered by MEDLINE, where they become difficult to locate. Moreover, the fact that many reports are intended for national journals published in languages other than English creates an additional language barrier for many researchers. For these reasons, hand-searching of national peer-reviewed journals may be necessary in order to maximise the comprehensiveness of a literature search of OH interventions.

    To evaluate the role of hand-searching, we systematically screened reports published in Italian language peer-reviewed scientific journals to find evaluations of OH interventions. Then we calculated how many of the potentially relevant studies retrieved by hand-searching could not be found by searching PubMed with optimised search strategies.3 ,8

    Methods

    We set out to identify full original articles published in Italian language peer-reviewed journals between 1990 and 2008 that met the Cochrane Occupational Safety and Health Review Group's (OSHRG) classification criteria for OH intervention studies.9 We excluded reports that were published only in ad hoc collections of conference proceedings and other theme-based supplements. We additionally decided to exclude studies that went on to be reported in English language journals. However, we did consider articles from an Italian language journal, Prevenzione Oggi, which since 2005 has provided parallel English language versions, based on the rationale that these publications cannot be traced via PubMed. Identification of eligible reports was accomplished in two phases. In an initial screening phase, we retrieved all potentially eligible articles. Four of us (DP, MF, GM, AF) first hand-searched all articles published in the Italian language peer-reviewed scientific journals whose scope directly concerns the OH sector or preventive medicine. These include: Annali dell'Istituto Superiore di Sanità, Archivio di Scienze del Lavoro (extant until 1995), Epidemiologia e Prevenzione, Giornale degli Igienisti Industriali, Giornale Italiano di Medicina del Lavoro ed Ergonomia (until 1997, Giornale Italiano di Medicina del Lavoro), La Medicina del Lavoro, L'Igiene Moderna, Prevenzione Oggi, and Rivista degli Infortuni e delle Malattie Professionali. All articles published in these journals were considered, even those exceptionally published in languages other than Italian. For each article, we scanned title, aims, tables and figures and earmarked all potentially relevant reports.

    Each potentially eligible article was then submitted to a detailed independent reading by three of us (AB, SM, FZ) to evaluate whether any portion of the study fully met the OSHRG criteria for classification as an OH intervention study.9 To exclude studies subsequently reported in the English language, one of us (AF) scanned titles and abstracts of articles published in the English language that were written by either the first author or the last author of the studies published in Italian.

    Adopting the OSHRG criteria,9 two of the authors (AF, RMTC) classified the study design (A codes) of eligible studies as randomised controlled trial (RCT) or cluster RCT (A1), controlled trial or prospective cohort study (A2), time-series (A3), before–after comparison without a concurrent control group, quality of care study or comparison with arbitrary controls (A4).

    Studies were also categorised according to the OH outcomes (B codes,9 data not shown).

    To explore differences between hand-searching and PubMed, we compared numbers of relevant articles identified by hand-searching with those obtained by applying OSHRG's most sensitive and most specific PubMed search strings.3 ,8 The comparison considered only the three hand-searched Italian journals that were indexed in MEDLINE: Epidemiologia e Prevenzione; Giornale Italiano di Medicina del Lavoro ed Ergonomia; La Medicina del Lavoro.

    For the indexed articles, we calculated the number needed to read (NNR) as the number of articles identified by hand-searching or by the OSHRG's strings3 ,8 that on average was necessary to read to identify one OH intervention study.

    Results

    After hand-searching all the Italian language journals dedicated to OH or preventive medicine, we traced and retrieved 25 articles published only in the Italian language between 1990 and 2008 that met the OSHRG classification criteria for OH intervention studies. Since two articles reported results from more than one study design, we considered a total of 27 studies. The majority (52%, 13/25) of the articles were published in La Medicina del Lavoro (the other journals represented were: Archivio di Scienze del Lavoro, n=4; Giornale degli Igienisti Industriali, n=3; Epidemiologia e Prevenzione, n=2; Giornale Italiano di Medicina del Lavoro ed Ergonomia, n=1; L'Igiene Moderna, n=1; Prevenzione Oggi, n=1). The total number of articles published in the three Italian journals that were also indexed in MEDLINE during the study period was 4035. As expected, all the 16 articles on OH intervention studies retrieved by hand-searching among these three journals were indexed in MEDLINE. Hence, we could calculate a NNR for hand-searching of 252 (4035/16). Table 1 reports the numbers of studies retrieved by hand-searching and by applying the OSHRG's electronic search strategies.3 ,8 Applying the most sensitive search strategy in PubMed we retrieved all 16 relevant articles among the total yield of 2112 references, thus giving a NNR of 132 (2112/16). The most specific search strategy on the other hand identified 9 (56%) articles among 537 references, resulting in a NNR of 60 (537/9). A brief description of all retrieved OH intervention studies is provided in Web Appendix 1.

    View this table:
    Table 1

    Distribution of Italian occupational health intervention articles according to study designs

    Discussion

    To explore the need for hand-searching when looking for OH intervention studies published in languages other than English, we hand-searched the full reports of effectiveness studies published in Italian language scientific journals between 1990 and 2008. Our findings suggest that a sensitive electronic search strategy applied to PubMed should be able to identify the vast majority of the Italian articles on OH intervention studies indexed in MEDLINE. Furthermore, when applying OSHRG's most sensitive search strategy without any further limitation, the NNR almost halved compared with hand-searching, permitting a consistent saving of time.

    When we applied the OSHRG's most specific search strategy,3 ,8 we were unable to retrieve a considerable proportion (>40%) of the OH intervention studies identified by hand-searching. However, the performance of the most specific search strategy in terms of NNR confirms that this strategy should be considered as a valuable tool for first-line bibliographic searches.

    The vast majority (81%) of the identified OH intervention studies had a before–after design, whereas the proportion of before–after studies in English language OH journals is only 44%.10

    Table 1 shows that all the effectiveness evaluations performed with a study design other than before–after (ie, RCT, non-randomised controlled trials and interrupted time-series) were described in articles indexed in MEDLINE. This observation could suggest that the coverage of electronic search strategies is greater when looking only for high quality intervention studies. Our results are in line with those of Rollin et al,11 who estimated that MEDLINE allows access to about 90% of the high-quality OH intervention studies included in the reviews of the Cochrane Library indexed under the topic ‘occupational health field’ in December 2009.

    The evaluation of grey literature was outside the scope of this overview. The OH intervention studies reported in the proceedings of the Italian Society of Occupational Medicine and Industrial Hygiene congresses between 1988 and 2003 have been reviewed elsewhere:12 of the 108 studies retrieved only six were controlled trials, none of which was randomised (no interrupted time-series analysis was found). Only five of the 108 studies presented in the proceedings were subsequently reported in a peer-reviewed journal indexed in MEDLINE.12

    Conclusions

    For journals indexed in MEDLINE, searching PubMed with a sensitive and optimised search strategy3 will locate most of the OH intervention studies, even if published in a language other than English. However, when conducting particularly in-depth research (eg, a systematic review on a little studied intervention), hand-searching of important national journals not indexed in MEDLINE should be considered to maximise coverage of the search.

    References

    1. ↵
      1. Verbeek JH,
      2. van Dijk FJ,
      3. Malmivaara A,
      4. et al
      . Evidence-based medicine for occupational health. Scand J Work Environ Health 2002;28:197–204.
      OpenUrlPubMedWeb of Science
    2. ↵
      1. Verbeek J,
      2. Husman K,
      3. van Dijk F,
      4. et al
      . Building an evidence base for occupational health interventions. Scand J Work Environ Health 2004;30:164–70.
      OpenUrlPubMedWeb of Science
    3. ↵
      1. Verbeek J,
      2. Salmi J,
      3. Pasternack I,
      4. et al
      . A search strategy for occupational health intervention studies. Occup Environ Med 2005;62:682–7.
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Gehanno JF,
      2. Rollin L,
      3. Le Jean T,
      4. et al
      . Precision and recall of search strategies for identifying studies on return-to-work in Medline. J Occup Rehabil 2009;19:223–30.
      OpenUrlCrossRefPubMed
    5. ↵
      1. Mattioli S,
      2. Zanardi F,
      3. Baldasseroni A,
      4. et al
      . Search strings for study of putative occupational determinants of disease. Occup Environ Med 2010;67:436–43.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Smith DR
      . Identifying a set of ‘core’ journals in occupational health, Part 1: lists proposed by others. Arch Environ Occup Health 2010;65:106–10.
      OpenUrlCrossRefPubMed
    7. ↵
      1. Smith DR
      . Identifying a set of ‘core’ journals in occupational health, Part 2: lists derived by bibliometric techniques. Arch Environ Occup Health 2010;65:173–5.
      OpenUrlCrossRefPubMed
    8. ↵
      Occupational Safety and Health Review Group. Criteria for Occupational Health Intervention Studies. http://osh.cochrane.org/criteria-occupational-health-intervention-studies (accessed 19 Dec 2011).
    9. ↵
      Occupational Safety and Health Review Group. Search Strategies. http://osh.cochrane.org/search-strategies (accessed 19 Dec 2011).
    10. ↵
      1. Ruotsalainen JH,
      2. Verbeek JH,
      3. Salmi JA,
      4. et al
      . Evidence on the effectiveness of occupational health interventions. Am J Ind Med 2006;49:865–72.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Rollin L,
      2. Darmoni S,
      3. Caillard JF,
      4. et al
      . Searching for high-quality articles about intervention studies in occupational health—what is really missed when using only the Medline database? Scand J Work Environ Health 2010;36:484–7.
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Mattioli S,
      2. Baldasseroni A,
      3. Mancini G,
      4. et al
      . [Effectiveness studies in the Italian Society of Occupational Medicine and Industrial Hygiene congresses (1989–2003)]. Med Lav 2005;96:513–48.
      OpenUrlPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Funding The study was supported by INAIL (Istituto Nazionale Assicurazione contro gli infortuni sul Lavoro) Direzione Regionale Emilia-Romagna, Bologna, Italy; Regione Emilia-Romagna (Emilia-Romagna Regional Administration), Bologna, Italy; and the University of Bologna.

    • Competing interests None.

    • Provenance and peer review Not commissioned; externally peer reviewed.