Elsevier

The Spine Journal

Volume 6, Issue 1, January–February 2006, Pages 72-77
The Spine Journal

Technical Review
An Independent AGREE Evaluation of the Occupational Medicine Practice Guidelines

https://doi.org/10.1016/j.spinee.2005.06.012Get rights and content

Abstract

Background and context

A large number of practice guidelines are being produced by numerous organizations. Health-care professionals need to critically evaluate these practice guidelines to understand whether they are well constructed and representative of the preponderance of evidence. The guideline development process should be precise and rigorous to ensure that the results are reproducible and not vague.

Purpose

To evaluate the quality of the second edition of the practice guidelines published by the American College of Occupational and Environmental Medicine (ACOEM Guidelines).

Study design/setting

Four appraisers used the AGREE (Appraisal of Guidelines Research and Evaluation) guideline evaluation instrument to evaluate the ACOEM Guidelines.

Methods

The Guidelines were evaluated with the AGREE guideline evaluation instrument. The AGREE instrument has been widely adopted around the world, and the authors recommended that it be adopted as the standard of guideline construction process evaluation in the United States. The instrument standardizes the quantitative assessment of quality for a guideline's development process across six domains that include: scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, application, and editorial independence. Scores from four assessors were collected and interpreted. Additionally, each evaluator selected one of four global assessment choices: “strongly recommended for use in practice,” “recommended for use with some modification or proviso,” “not recommended as suitable for use in practice,” or “unsure”.

Results

The ACOEM Guidelines scored highest in the dimensions that evaluated reporting of the guideline's scope and purpose (79.63) as well as clarity and presentation (86.81). The guideline scored much lower in the remaining areas that included stakeholder involvement (46.06), rigor of development (26.59), application (31.48), and editorial independence (19.17). The global assessment was unanimous with all four evaluators assessing the guideline as recommend with proviso.

Conclusions

Many of the Guidelines recommendations were consistent with current literature and guidelines; however, the AGREE assessment instrument evaluates the guideline development process and not the content. All the evaluators thought the content of the guidelines was substantially better than the documentation of the guideline construction process. The ACOEM Guidelines appear to have content consistent with their stated objectives, but the reporting of the guidelines construction process, particularly the rigor of recommendation development, is flawed, and the recommendations may not be valid owing to possible evidence selection deficiencies. The reader should consider these flaws and limitations when using the guideline. The reader should consider utilizing guidelines of higher quality when possible. Future guidelines should incorporate better reporting and give closer attention to guideline construction.

Introduction

The information age has brought us technology which allows the sharing of data quickly and easily around the world. A physician now has the option of obtaining information regarding clinical practice management from many sources at any time.

It has long been the purpose of practice guidelines to organize the huge amount of relevant raw data for easier understanding and application in a clinical setting. Today, a physician can access a number of guidelines pertaining to the same condition or topic. Often, recommendations among guidelines vary, and in some cases they are conflicting. The question of how to select the best clinical guideline then arises. Health-care professionals need to critically evaluate guidelines to understand whether they are well constructed and representative of the preponderance of evidence. The guideline development process should be precise and rigorous to ensure that the results are reproducible and not vague [1], [2], [3].

Clinical practice guidelines impact the manner in which patient care is assessed by peer review and often serve as a basis for payer decision making regarding the delivery of spine care to patients. The American College of Occupational and Environmental Medicine (ACOEM) Guidelines have been adopted by several compensation systems as a standard for evaluation and management of work injuries. In California, where low back injuries comprised 39% of new worker's compensation claims in 2003 [4], the ACOEM Guidelines are legislatively mandated as “presumptively correct” for evaluation and management of all musculoskeletal injuries in the worker's compensation system.

Section snippets

Purpose

It is generally accepted that peer review and testing of health-care guidelines should be performed prior to their acceptance as being valid and their subsequent utilization in clinical practice [5], [6]. The purpose of this paper is to report the results of an analysis of the quality of the Occupational Medicine Practice Guidelines, Second Edition, published by the ACOEM [7]. The analysis employed the standardized AGREE (Appraisal of Guidelines Research and Evaluation) instrument to assesses

Study design/setting

Four appraisers with prior guideline evaluation experience volunteered to rate the ACOEM Guidelines using the AGREE instrument. All the participants have advanced training in the critical assessment of clinical data, and all are familiar with the AGREE instrument. Three are chiropractic physicians, one is a medical physician. One has a PhD, one a master's degree. Three appraisers have advanced certification in orthopedics, one in occupational and environmental medicine as well as in emergency

Instrument

The AGREE guideline evaluation instrument was selected as it has been assessed as reliable and valid for the purpose of guideline evaluation [10]. The AGREE instrument is a generic tool developed by the AGREE Collaboration in 1998. It was designed for the European Union under the Biomedicine and Health Research (BIOMED 2) Program to help guideline developers and users assess the methodological quality of clinical practice guidelines. Several additional countries, including Canada, have since

Results

Inter-examiner agreement for item scores was very high for the medical and chiropractic assessors. For most items across all chapters, agreement was 100%. When disagreement in the form of different item scores occurred, the disagreement was in degree of criterion satisfaction rather than whether or not the criterion had been satisfied. The medical reviewer's item score was consistently within the 95% confidence limits of the mean of the three chiropractic evaluators for that item when there was

Conclusions

The ACOEM Guidelines, Second Edition appears to have content consistent with their stated objectives, but the reporting of the guidelines construction process, particularly the rigor of recommendation development, is flawed, and the recommendations may not be valid because of possible evidence selection deficiencies. The reader should consider these flaws and limitations when using the guidelines. The reader should consider using guidelines of higher quality when possible. Future guidelines

References (21)

There are more references available in the full text version of this article.

Cited by (55)

  • Workplace mental health: An international review of guidelines

    2017, Preventive Medicine
    Citation Excerpt :

    Exploring the development process of the recommendations has been found to be key to successful implementation (Hulshof and Hoenen, 2007). Several reviews have evaluated the quality of occupational guidelines and highlighted that a lack of consultation with relevant professional and population groups, a narrow review of the literature and a lack of extra resources to assist in the implementation of recommendations are common (Cates et al., 2006; Hulshof and Hoenen, 2007; Kinnunen-Amoroso et al., 2009; Manchikanti et al., 2008; Staal et al., 2003). These shortcomings may explain why employers and relevant health professionals often fail to engage with recommendations.

  • Improvement evident but still necessary in clinical practice guideline quality: a systematic review

    2017, Journal of Clinical Epidemiology
    Citation Excerpt :

    In accordance with previous reviews of CPG quality [28,31,42–44], the present review found that domain scores were lowest in the Applicability and Editorial independence. These areas of CPG development have historically been weak and continue to be an area in which great improvement is possible [10,11,28,31,43–45]. Guideline development groups can improve their domain scores in Editorial independence by providing transparent and comprehensive information regarding funding sources and including a direct statement declaring the presence or the absence of each author's conflicts of interest.

  • Appraisal of guidelines developed by the World Health Organization

    2014, Public Health
    Citation Excerpt :

    Delineation of guidelines' scope and purpose, and clarity of presentation were generally better done, however, stakeholder engagement, editorial independence, and applicability were often poorly reported and/or executed. Our findings are similar to those reported in other studies examining cohorts of guidelines using the AGREE II instrument9,15–22 including several cohorts of WHO guidelines.8,9 Scores for the domains of scope and purpose, and clarity of presentation tend to be higher across these various studies, while scores for stakeholder involvement, editorial independence and applicability are often low.

View all citing articles on Scopus

FDA device/drug status: not applicable.

Nothing of value received from a commercial entity related to this manuscript.

View full text