Article Text

Download PDFPDF

Definition and assessment of specific occupational demands concerning lifting, pushing, and pulling based on a systematic literature search
Free
  1. J Bos,
  2. P P F M Kuijer,
  3. M H W Frings-Dresen
  1. J Bos, P P F M Kuijer, M H W Frings-Dresen, Coronel Institute for Occupational and Environmental Health, Academic Medical Center, University of Amsterdam, Amsterdam, Netherlands
  1. Correspondence to:
 Drs J Bos, Coronel Institute for Occupational and Environmental Health, Academic Medical Center, University of Amsterdam, PO Box 22700, 1100 DE Amsterdam, Netherlands;
 jurriaan.bos{at}amc.uva.nl

Abstract

Aims: (1) To find a universal strategy for the identification of specific demands of a job or task, focusing on occupations in which there may be an increased risk for health complaints owing to these specific demands. (2) To select reliable and valid tests concerning lifting, pushing, and pulling, which consider the relation between occupational work demands and the assessment of the maximally acceptable load on an individual level.

Methods: Literature search was performed using Medline (1988 to May 2001), Embase (1966 to May 2001), and NIOSHTIC (1971–98).

Results: No universal strategy was found for the definition of specific occupational demands. Therefore a “three step strategy” was formulated for defining specific occupational demands in a job or a task in order to prevent health complaints on an individual level. Many tests were found in the literature concerning lifting, but only a few concerning pushing and pulling. None of the tests concerning pushing, pulling, or lifting considered the relation between work demands and the assessment of the maximally acceptable load on an individual level. Furthermore, none of the tests met the criteria of reliability and prognostic value for musculoskeletal complaints completely. Only for the prognostic value of relative strength capacity tests concerning pushing pulling and lifting, did there appear to be limited proof for the development of musculoskeletal complaints.

Conclusions: In general, for the prevention of work related health complaints, it can be suggested that more attention should be paid to: (1) the definition of specific occupational demands; (2) the assessment of specific occupational demands; and (3) the quality of tests for specific occupational demands.

  • specific occupational demands
  • tests
  • prevention
  • CV, coefficient of variation
  • DOT, Dictionary of Occupational Titles
  • EPIC, Employment Potential Improvement Corp
  • MMH, material handling activities
  • PILE, progressive isoinertial lifting evaluation
  • RAL, rating of acceptable load

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Despite the development of modern technology, extensive automation, mechanisation, and work related interventions, many occupations still require manual material handling activities (MMH) such as pushing, pulling, and lifting. Pushing, pulling, and lifting pose physical stresses to the individual. These are manifested as strains on the musculoskeletal and cardiovascular systems. If the strain causes physical overload, it may result in the development of discomfort, fatigue, or musculoskeletal injuries or disorders. Several studies have reported pushing, pulling, and lifting as risk factors of back, neck, and shoulder complaints.1–6 It is generally believed that the prevention of health complaints necessitates focus on specific risk factors of the job and implementation of preventive measures, such as engineering modifications (for example, workstations) or changes in the organisation of the work (for example, work/rest schedules or job rotation). However, despite existing preventive measures and the general knowledge of pushing, pulling, or lifting as risk factors, a universal strategy for the identification of specific demands of the job or task is needed for prevention of health complaints at the individual level. In summary, these specific occupational demands can be defined as work demands that are risk factors for the development of occupational health complaints with respect to a specific job.

When pushing, pulling, or lifting are identified as specific physical occupational demands, it is important to test the maximally acceptable load of the individual worker at risk, concerning these activities, for prevention of health complaints on an individual level. Tests concerning the specific occupational demands “lifting”, “pushing”, and “pulling” can be part of pre-employment testing or a periodic health surveillance for a specific job. For health surveillance, to assess the individual risk appropriately, consideration of the quality of tests is important. Tests for assessment of specific occupational physical demands should meet criteria of reliability, prognostic value for health complaints, and content validity. Content validity refers to the test’s ability to measure the daily performed occupational work demands. Tests concerning pushing, pulling, or lifting should consider both the work demands, for example, tasks and activities, as well as the maximally acceptable individual load. The first goal of this study is to identify and evaluate a strategy, focusing on occupations in which there may be an increased risk for health complaints owing to specific occupational physical demands. The second goal is to select tests concerning lifting, pushing, and pulling, which consider the relation between work demands and maximally acceptable load on an individual level, and are reliable and valid.

In summary the present literature study addresses the following questions:

  1. Is there a strategy for defining the specific occupational demands in a job or task?

  2. Which tests are available for the assessment of the individual risk to prevent health complaints resulting from lifting, pushing, and pulling?

  3. Which tests consider the relation between work demands and the maximally acceptable load on an individual level (content validity)?

  4. What is the reliability (test-retest) and prognostic value for musculoskeletal complaints of these tests?

Main messages

  • In the literature no universal strategy is available for the definition of specific occupational demands in a job or a task in order to prevent health complaints on an individual level.

  • A total of 43 static or dynamic tests concerning lifting, pushing, and pulling were found. Of these tests, 22 are reliable and 22 are partly content valid. Twelve tests have limited prognostic value for back or other musculosketal complaints. However, none of the tests concerning pushing, pulling, or lifting considered the relation between work demands and the assessment of the maximally acceptable load on an individual level, and the criteria of reliability and prognostic value for musculoskeletal complaints completely.

Policy implications

  • Information on (1) exposure to work demands, (2) existing knowledge of epidemiological studies concerning risk factors for health complaints, and (3) workload guidelines, should be used to identify “specific occupational demands” for specific jobs.

  • Occupational health physicians, researchers, or other users of tests concerning pre-employment testing or periodic health surveillance should pay more attention to: (1) the definition of specific occupational demands; (2) the assessment of specific occupational demands; and (3) the quality of tests for specific occupational demands.

METHODS

A systematic literature search

Two systematic literature searches were performed on: (1) definition of specific occupational demands; and (2) assessment of the individual risk concerning lifting, pushing, and pulling. These searches will be elaborated below.

Search 1: Definition of specific occupational demands

To find a strategy for the definition of specific occupational demands a literature search was performed using Medline (1988 to May 2001), Embase (1966 to May 2001), and NIOSHTIC (1971–98). Combinations of the following keywords were used to identify relevant studies: occupational demands, physical demands, specific, job, task, risk factors, external exposure, occupational diseases, musculoskeletal diseases, prevention, strategy.

Inclusion criteria

The following inclusion criteria were applied:

  • Articles, reports, or books had to be written in Dutch or English.

  • Articles, reports, or books on a strategy for the definition of specific occupational physical demands had to consider exposure to occupational work demands. In this literature study work demands that are risk factors for the development of occupational health complaints are defined as specific occupational demands.

Search 2: Assessment of the individual risk concerning pushing, pulling, and lifting

The literature was searched for tests that assess the maximally acceptable load on an individual level, concerning pushing, pulling, and lifting. To select tests a literature search was performed in the same databases as mentioned previously. The following keywords were used: pushing, pulling, lifting, combined with muscle strength, tests, testing, assessment, evaluation, work capacity, functional capacity evaluation, mechanical (load), physical (load), energetic (load), workload, psychophysical, low back complaints, low back pain, neck pain, shoulder pain, guideline, validity, reliability, prognostic value. Furthermore, experts in the field were asked for information or literature on the assessment of the specific occupational physical demands concerning pushing, pulling and lifting. The “snowball” method was used to identify relevant references in the articles and reports retrieved. According to the snowball method, the references of each report or article found by the electronic search were searched manually for more articles on tests concerning lifting, pushing, and pulling meeting the inclusion criteria.

Inclusion criteria

The following inclusion criteria were applied:

  • Reports, articles, or books had to be written in Dutch or English.

  • Reports, articles, or books on assessment of the individual risk concerning pushing, pulling, and lifting had to describe tests for the assessment of individual maximally acceptable load, concerning these activities.

Evaluation of the available information on tests

All available information on (1) consideration of the relation between work demands and maximally acceptable load in the tests (content validity), (2) reliability of tests, and (3) prognostic value of tests concerning pushing, pushing, and lifting was gathered and classified according to the mode of testing—that is, static or dynamic testing. In static testing the person performs a maximal external force against a stationary object. In dynamic testing the person has to move an object over a certain distance. For the evaluation of the available information on selected tests the following were successively considered.

  1. Content validity. The evaluation could be:

    • The test considers the relation between work demands, in terms of duration, frequency, and intensity of lifting, pushing, or pulling activities for the specific occupation, and maximally acceptable load. Duration, frequency and intensity of occupational activities are assessed during the test and are extrapolated to available information on these work demands during a normal working day (+).

    • The test considers the relation between work demands and individual maximally acceptable load only partly (+/−).

    • The test does not consider the relation between work demands and individual maximally acceptable load (−).

  2. Assessment of the (test-retest) reliability of the tests. The evaluation could be:

    • The test is (test-retest) reliable, indicated by a coefficient of variation (CV) and/or (intraclass) correlation (+).

    • The test is (test-retest) reliable only for some test components (+/−), indicating that the CV or intraclass correlation was only calculated for some components.

    • The test is not reliable (test-retest), indicated by a CV and/or (intraclass) correlation (−).

    • No studies have been found on (test-retest) reliability of the tests meeting the keywords in the literature search (?).

  3. Assessment of the prognostic value of the tests. The evaluation could be:

    • The test has prognostic value for the development of musculoskeletal complaints and the body regions of the musculoskeletal complaints are specified further. In the study the prognostic value has been proven by calculation of an odds ratio or a relative risk (+).

    • The test only partly has prognostic value for the development of musculoskeletal complaints. The body regions of the complaints are not specified further, the prognostic value of the tests has not been studied/proven for all test components. In the study the prognostic value has been proven by calculation of an odds ratio or a relative risk (+/−).

    • The test has no prognostic value for the development of musculoskeletal complaints indicated by an odds ratio or a relative risk (−).

    • No studies have been found on prognostic value of the tests (?).

RESULTS

Search 1: Definition of specific occupational demands

The literature search in the various databases and the interviews did not result in relevant articles on the definition of specific occupational physical demands, which consider the exposure to specific occupational work demands. Therefore, no universal strategy was found for the definition of specific occupational demands.

Search 2: Assessment of the individual risk concerning lifting, pushing, and pulling

The literature search in the databases and the interviews resulted in 55 relevant articles, two reports, and two books on tests concerning pushing, pulling, and lifting. One static7–33 and several dynamic tests32–62 were found in the literature for the assessment of the individual maximally acceptable load concerning lifting. Only a few tests were found on the assessment of the maximally acceptable load concerning pushing and pulling statically11,12,63 or dynamically.19 Table 1 presents a review of the articles, reports, and books indicating (1) the relation between work demands and the maximally acceptable load, (2) the reliability of the tests, and (3) the prognostic value of the tests.

Table 1

Relation between work demands and the assessment of the maximally acceptable load on an individual level, reliability, and prognostic value for health complaints of available tests concerning lifting, pushing, and pulling

Static strength testing concerning lifting, pushing, and pulling

The static lifting test comprises the assessment of the maximal isometric lifting strength, and all the tests concerning pushing and pulling are used to assess the maximum isometric push and pull strength. By this method, the person performs a maximal external force against a stationary object in a vertical upward (lifting) or horizontal (pushing or pulling) direction. Several tests were found which can be used to assess the maximal isometric strength.7–39

Dynamic strength testing concerning lifting, pushing, and pulling

Dynamic lifting tests

In dynamic lifting a person has to move an object vertically over a certain distance. Dynamic lifting tests assess the maximal lifting weight, the maximal lifting power, the maximal number of repetitions of lifting, or the individual rating of maximally acceptable lifting loads for an eight hour working day. Examples of dynamic lifting tests are the operational lifting test of Jacobs and colleagues,38 the progressive lifting test,38 the Ariel,36–39 the “repeated lifting test”,37 and the Aristokin37 (see table 1). The method Isernhagen Work System (IWS),32–34 the method Blankenship,32–34 the Ergos Worksimulator,34–36 and the Ergo-kit33–35 have different protocols for the assessment of the maximal individual lifting weight.

Dynamic pushing and pulling tests

Four tests, protocols of the methods IWS32–34 and Blankenship,34,35 the push/pull test of Laflin and colleagues,19 and the dynamic push/pull test of the physical work performance evaluation (PWPE)58 were found on dynamic pushing or pulling (see table 1). The tests of the IWS32–34 and Blankenship tests34,35 are not used to assess the maximum dynamic strength. These tests assess the ability to push and to pull dynamically for a short period of time. In the test of Laflin and colleagues,19 the amount of push or pull force needed to move a sled filled with boxes is measured with a dynamometer.

Content validity

Static tests

With the assessment of the maximal isometric strength concerning lifting, pushing, and pulling, the relation between work demands and the assessment of the maximally acceptable load on individual level is only partly considered. The reason for this is that isometric strength can be related to externally performed lifting forces, but not to duration and frequency of lifting, pushing, or pulling.

Dynamic tests

Using the Dictionary of Occupational Titles (DOT),35,41,42 some dynamic lifting tests only partly consider the relation between work demands and the maximally acceptable load on individual level. The DOT35,41,42 describes the physical demands for a wide variety of jobs in the USA. In fact, maximally acceptable lifting weight is established by consideration of duration, frequency, and intensity of lifting tasks for occupations in the USA (as prescribed in the DOT35,41,42) only. The other tests do not consider the relation between work demands and the assessment of maximally acceptable load on individual level.

The push/pull test of the PWPE considers the relation between dynamic push and pull demands and the maximally acceptable push and pull load only partly by using the DOT.35,41,42

Reliability

Isometric strength assessment tests concerning lifting, pushing, and pulling are reliable.15 Only some of the dynamic tests concerning lifting, pushing, and pulling, such as the Ariel,37–39 the Aristokin,43 the Lido lifting test,17 the Employment Potential Improvement Corp (EPIC) lifting test,50–52 the lifting test according to the progressive isoinertial lifting evaluation (PILE) protocol,24,43,45,49,55,56 the rating of acceptable load (RAL) lifting test,57 the incremental lifting machine,26,30,32,53,54 the Lifttest of Laflin and colleagues,19 the LIFTEST,20,31 the six days functional capacity evaluation,59 and the one repetition maximum method61 have been found to be reliable. For the other lifting tests no studies were found on (test-retest) reliability of the assessment of maximal lifting weight.

Prognostic value

There is limited epidemiological proof of the relation between externally performed forces and individual maximally performed isometric strength (the individual relative strength capacity) concerning lifting, pushing, and pulling as a predictor for the development of back complaints and other musculoskeletal complaints.7–15

None of the dynamic lifting tests have been examined regarding their prognostic value for musculoskeletal complaints. Only one study12 has shown evidence for the relation between the maximally performed external forces in the job and the maximally performed isometric strength for pushing on trunk height and pulling on shoulder height and the development of musculoskeletal complaints.

DISCUSSION

Search 1: Definition of specific occupational demands

In the literature no strategy was found for the definition of specific occupational demands. One Dutch guideline for pre-employment testing in the Netherlands64 gives an overview of some specific occupational demands, the content of the medical examination, and criteria for medical fitness for occupations. The Dutch guideline defines specific occupational demands as “work demands that constitute an increased risk for developing health complaints”. The Dutch guideline is not completely satisfactory for all occupations, because it does not consider the specific tasks and activities and specific health complaints of many occupations. To define specific occupational demands, scientific knowledge is needed, which clarifies the relation between the exposure to work demands and the specific health complaints. In order to acquire this knowledge, the following “three step strategy”65 can be applied to the specific occupational setting:

  1. Describe the exposure to work demands for the occupation, in terms of duration, frequency, and intensity. Information on the exposure to work demands may explain the risk of developing health complaints.66

  2. Establish the risk of work demands for health complaints on the basis of epidemiological studies. For specific occupations the incidence of health complaints can be assessed, as well as the relative risk of tasks and activities for development of musculoskeletal complaints.

  3. Establish whether work demands exceed existing workload guidelines. These guidelines prescribe the maximally acceptable external exposure to work demands in terms of duration, frequency, and intensity for an eight hour working day or shorter working periods.

Tasks and activities, such as pushing, pulling, and lifting, can be defined as specific occupational demands when there is an increased risk for health complaints and workload guidelines are exceeded. In summary, to formulate the specific occupational physical demands this “three step strategy” has to be applied to data on duration, frequency, and intensity of work tasks and activities of the specific occupation. Unfortunately data on duration, frequency, and intensity of pushing, pulling, and lifting are not available for many occupations. For several occupations in the Netherlands, such as refuse collectors,67 lorry drivers,68 train stewards, and workers in nursing homes, flower auctions, and postal services,69 data on pushing, pulling, and lifting are available. These data can thus be compared to guidelines64,65 to establish whether pushing, pulling, and lifting can be defined as specific occupational demands. Although the guidelines on lifting, pushing, and pulling70–78 are formulated in terms of intensity (the maximal weight or force) only, they do consider duration and frequency of lifting, pushing, and pulling activities and some specific work factors. For example, hand height and walking distance concerning pushing, pulling, or lifting, are work factors that are considered in these guidelines.

Search 2: Assessment of the individual risk, concerning pushing, pulling, and lifting

Content validity

A remarkable finding of this study is that only a few tests concerning pushing, pulling, or lifting, consider content validity in terms of the consideration of duration, frequency, and intensity of occupational work tasks and activities. In fact, in some tests the DOT35,41,42 is used to assess the maximal acceptable load concerning the specific occupation. In these tests the worker must at least have the physical capacities equal to the physical demands of the job as prescribed by the DOT. The DOT has defined the majority of jobs in the USA according to 20 job factors and has prescribed the work demands of the jobs. However, the validity of the DOT is questionable because the physical demands of the DOT are not based on quantitative work task analyses but are based on based on subjective consensus meetings of experts in the field. In addition, “job exposure matrices” are defined. Several studies indicate that the sensitivity and specificity of job exposure matrices is low.80 Therefore, a “task analysis” is recommended to get more insight into the duration, frequency, and intensity of tasks and activities performed in a specific job.

The maximally acceptable load during the test might not be the same as the maximally acceptable load concerning the specific occupational setting, because lifting, pushing, and pulling activities are always performed for longer periods in reality. Therefore, many experts believe assessments of the maximally acceptable loads should not only include the maximum load that can be lifted, pushed or pulled safely, but also safe limits for loads that are handled occasionally and frequently.59 Occasionally and frequently are operationally defined as performance of repeated lifting, pushing, or pulling activities up to 33% and between 33% and 67% of the average workday, respectively.59 Several studies have developed models30,59 that can be used to extrapolate the test results concerning assessments of infrequent lifting to the maximally acceptable lifting weight concerning frequent lifting. Only in some tests33–36 concerning pushing, pulling, and lifting is the maximally acceptable load for longer work periods related to the occupational work demands (duration, frequency, and intensity of lifting, pushing, and pulling activities) for assessment of an increased risk for musculoskeletal complaints on an individual level. Furthermore, in this study several tests were found concerning non-frequent lifting or lifting for short duration, and several tests concerning frequent lifting and/or carrying for a longer duration. For example, in a majority of industrial jobs lifting is often done for a longer duration. If lifting for a longer duration is a work demand, it might be better to simulate lifting or carrying for a prolonged period in the test.

Sometimes certain tasks in certain jobs cannot be redesigned in order to reduce the risk of health complaints. In that case the employer should define specific occupational demands. The tests concerning lifting pushing and pulling, as part of pre-employment testing and periodic health surveillance, should then be used to protect the individual worker at risk. As a result equal job opportunities for employees are not always possible. Thus, not only the results of the tests for specific occupational demands should be considered, but also the impairments of the employee for the specific job.

Content validity of tests also resembles simulation of body posture and speed of limb movement during performance of these activities in the tests. The static or dynamic tests, as found in the present literature study, do not always consider this simulation. Dynamic tests concerning lifting are subdivided into isoinertial or isokinetic tests in the literature.30,31 In isokinetic testing, the body segment velocity should remain constant during the maximal voluntary contraction. Isoinertial testing is a method of measuring the maximum weight a person is willing to handle, at his/her own selected speed within a specified range of movements. A disadvantage of both isometric and isokinetic methods concerning pushing, pulling, and lifting is that the actual performed movements are never isometric (static) or isokinetic (with a constant speed) in reality. Furthermore, several studies indicated that the results of isometric and/or static tests correlate poorly with performance of functional activities.32,33 These studies suggest that isoinertial tests concerning lifting might be preferred to isometric and/or isokinetic tests.

Reliability

Only a few of the selected tests for pushing, pulling, or lifting were studied on test-retest (or intrarater) reliability, which refers to the stability of the test scores. To enlarge the reliability of the assessment of maximally acceptable load the test should be performed repeatedly at least two times.24 The CV is frequently used by test leaders to determine whether a person is giving a consistent effort during testing, which is based on the assumption that an intentionally submaximal effort will result in a greater variability. Although there is much variability in the CVs, which different investigators report as acceptable (ranging from 5% to 29% in the peer reviewed literature7) for isometric and isokinetic strength testing, various studies indicate that the variation between repeatedly performed measurements during testing should not exceed 10–15% for strength testing.15 The exceeding of this variation might not be an accurate measure of reliability. In fact, the maximally acceptable variation is largely dependent on the test parameters, the variation of the test parameters from day to day (biological variation), the mode of testing, the type of physical activities performed in the test, and the test conditions. Furthermore, individuals with impairments may have higher CVs. Intraclass correlation is often indicated to be a statistic that reflects reliability more accurately than the coefficient of variation. Furthermore, there is no evidence that inconsistent response on isometric testing indicates submaximal performance.28,81 To assess the maximally acceptable load during the test, other indicators of sincerity of effort, such as percentage increase in heart rate or systolic blood pressure, are also used in several tests concerning lifting, pushing, or pulling. It should be noted that even if these indicators are within acceptable limits, maximal performance of the individual could never be guaranteed. This might be explained by the fact that these indicators are not related to the maximal individual strength directly.

Reliability also refers to the ability to achieve simular scores on an evaluation when administered by different evaluators (interrater reliability). This has not been studied further in this review.

Prognostic value

Few of the selected tests for pushing, pulling, or lifting considered prognostic value. The methodological quality of the studies on prognostic value of the tests was low. Firstly, musculoskeletal complaints were not specified further in two of the eight identified studies7,12 on prognostic value for musculoskeletal complaints. Secondly, three studies had a follow up period of approximately one year. One year might be too short to study the development of musculoskeletal complaints because the percentage of “drop outs” in some of the studies was high.82 Thirdly, none of the studies used a regression analysis or odds ratio to describe the prevalence of musculoskeletal complaints. Only one study14 calculated a relative risk of 1 to 1.2 of reporting back complaints from isometric lifting strength during a four year follow up period. Probably, the most likely reason why limited research has been done on the prognostic value of the tests for development of complaints is that this type of longitudinal research is difficult to administer, expensive, and takes time. Often tests for lifting, pushing, and pulling, as have been identified in this study, are used in sports, rehabilitation, insurance medicine for the purpose of training, rehabilitation, or functional capacity evaluation of individuals with health complaints. Until now, the prognostic value of these tests has received little attention in these areas.

Biomechanical, physiological, and psychophysical models23,59,67,70–78 have been developed in many studies to better understand the relation between performance of lifting pushing and pulling and the development of physical overload of body structures and/or musculoskeletal complaints, and to develop guidelines on maximal acceptable loads concerning lifting, pushing, and pulling. These guidelines prescribe maximal exposure limits on group level. Therefore, these guidelines cannot be used for assessment of the maximal acceptable load on an individual level. Furthermore, many investigators have adopted the psychophysical approach to determine the maximally acceptable weight of lifting using isometric18,21,38 and isoinertial38 strength tests concerning lifting as predictors.

Tests for the assessment of specific occupational physical demands should not only meet criteria of content validity, reliability and prognostic value for health complaints, but should also be safe and practical to administer.79 The isometric method has been reported as practical and safe.15 Moreover, the tests should contain a good sensitivity and specificity. No information is yet available on the sensitivity and specificity of the tests.

Overall it can be concluded that no strategy was found for defining specific occupational demands in a job or a task in the present study. Therefore a three step strategy was formulated for the prevention of health complaints on an individual level. Furthermore, in the present study many tests were found for the assessment of individual maximally acceptable loads concerning lifting. Some tests were found concerning pushing and pulling. None of the tests concerning lifting, pushing, and pulling considered the relation between work demands and the assessment of the maximally acceptable load on individual level and met criteria of reliability and prognostic value for musculoskeletal complaints completely.

In general, for the prevention of work related health complaints on an individual level, it can be suggested that more attention should be paid to:

  1. The definition of specific occupational demands.

  2. The assessment of specific occupational demands.

  3. The quality of tests for specific occupational demands.

REFERENCES