This is the second of two papers on a participative intervention aimed at reducing adverse psychosocial factors at work and at preventing mental health problems associated with work among healthcare providers. The first paper* documents the development and the implementation phases of the intervention. The current paper presents the results of the effectiveness phase of the intervention at 12 months.

In the past decade, the healthcare sector in the province of Québec has undergone significant organisational restructuring in order to reduce healthcare costs and improve efficiency. During this restructuring, there was a considerable increase in the prevalence of high psychological demands and the combination of high demands and high or low decision latitude among nurses.¹ Moreover, the prevalence of psychological distress and emotional burnout,² as well as the incidence of short term sick leave, and long term medically certified sick leave³ among nurses, were associated with job strain (a combination of high psychological demands and low decision latitude). In one of these studies, nurses exposed to job strain reported 60% more psychological distress, and nurses exposed to low social support at work 20% more distress than nurses not exposed to these psychosocial factors.¹

An increasing interest in stress prevention has led to many studies on preventive interventions aimed at reducing adverse psychosocial work factors and their impact on health; but the effectiveness of few of these has been evaluated.^4–6 Furthermore, evaluation research of preventive interventions often suffers from important methodological limitations.^5–7 For instance, Van der Hek and Plomb observed an inadequate follow up period in most available studies (less than one year) and the lack of a control group.⁷ The present research intended to avoid these limitations.

The fact that our research efforts have been supported since 1997 by a steering committee favoured the development of this project. The committee members represented major decision makers in the health network: the provincial Health Ministry, nurses’ professional order and union associations, the nursing department, and human resources management, as well as nurses from various local hospitals.

The theoretical concepts that provide the basis for our study are Karasek’s demand-control-support model⁸ and Siegriest’s effort-reward imbalance model.⁹ These models identify four psychosocial factors at work whose effects on physical and mental health are the most frequently documented.^10,11

We hypothesise that implementing an intervention targeting the psychosocial job environment and integrating care providers’ participation in an experimental hospital should allow a decrease in four adverse psychosocial factors, namely high psychosocial demands, low decision latitude, low social support, and effort-reward imbalance (intermediate effects), as well as a decrease in mental health problems at work (final effects).

The objectives of the study are:

• To describe the evolution of psychosocial factors at work and health problems between pre-intervention and post-intervention measures (12 months after) in the experimental hospital and in the control hospital (before and after, intra-group measure)

• To compare the changes observed between the pre- and post-intervention measures for each psychosocial factor and health problem in the experimental hospital compared with the control hospital (post-intervention inter-group measures).

METHODS

Study design and population

The research design is a before and after quasi-experimental type with a control group. The intervention was defined as the changes undertaken by the hospital to reduce adverse job psychosocial factors. Solutions proposed by an intervention team (IT) and adopted by the nursing department (NS) as well as any other objective change introduced with the explicit goal (or actual consequence) of improving one of the four targeted psychosocial factors was considered part of the intervention (for more details on the intervention, see Bourbonnais et al in this issue of the journal). The study population is comprised of care providing personnel in the experimental and control hospitals, both of which offer general and specialised short term care. The steering committee members and the research team combined their knowledge of the healthcare network operations to select the experimental and control hospitals. Criteria for selection were: their comparability in terms of size of hospital setting and type of health care provided (acute care), the potential commitment of management and union to support the research, the relative stability of staff turnover, and anticipated organisational changes over the research period. The population includes all care providers in direct contact with patients (nurses, orderlies, and auxiliary nurses), who occupy permanent full time or part time and temporary positions, or who are on call. Care providers on sick leave for more than three months and those working only two days per week over the three months preceding the pre-intervention or baseline measure were excluded from the study.

Data compiled from the questionnaire (before and after intervention measures)

Exposure to psychosocial job factors and health status of care providers were measured before the intervention as a prior risk evaluation and 12 months after the beginning of the intervention in order to evaluate its short term effects. These measures were carried out in both the experimental and control hospitals. The before and after intervention measures were done with the same validated instruments which measure the four targeted psychosocial factors as well as mental health indicators (psychological distress, professional burnout measured by two self-reported indicators of mental health).

The list of all caregiver personnel of both hospitals was obtained by the research team in order to reach the participants who were contacted at home and informed of the study objectives and modalities of their participation. A 30 minute telephone interview relating to psychosocial job factors and health was conducted by a firm specialised in telephone survey. Neither the firm nor the telephone interviewers knew which subjects were in the experimental or in the control hospitals. A research assistant regularly listened to the interviews in real time throughout the 2000 and 2002 proceedings to verify whether the questions were asked as instructed to limit a potential bias from inter-interviewer variation and ensure the quality of the compiled data.

Psychological demands (quantity of work, intellectual requirements, time pressure) and control or decision latitude (use and development of skills, control over work which implies latitude at work and participation in decisions) were evaluated using 18 items from Karasek’s job content questionnaire (JCQ).^12,13 The validity of the JCQ has been assessed in national population based surveys in the United States.⁸ The psychometric qualities of the French version of this instrument have been demonstrated.^14,15 Social support at work was measured by eight items from the JCQ assessing three components of support from supervisors and colleagues: socio-emotional support or esteem, which is of a socio-psychological or interpersonal nature; instrumental support, which measures extra resources or assistance with work tasks; and a negative level of support, hostility, or conflict.¹³ Reward at work (esteem, respect, job security, remuneration, and career opportunities) was measured by 11 items from Siegrist’s original instrument for which factorial validity and internal consistency have been documented.^16–18 In our data, the effort/reward imbalance was defined as a ratio of psychological demands on reward greater than 1 as recommended by Siegrist.¹⁶ In our study, internal consistency based on Cronbach’s coefficient alpha was 0.71 for job decision latitude, 0.76 for psychological demands, 0.81 for social support, and 0.77 for reward.

Psychological distress was measured using an abridged version (14 items) of a validated instrument, the Psychiatric Symptom Index (PSI).¹⁹ This instrument measures the frequency of symptoms of anxiety, depression, aggressiveness, and cognitive problems during the previous week. The French version of the PSI was validated in a Québec health survey.²⁰ PSI has sound concomitant validity with respect to four other measurements of mental health: health professional consultation for a mental health problem, hospital admission for this type of problem, presence of suicidal tendencies or suicide attempts, and consumption of psychotropic medication.²¹ In our data, the Cronbach’s alpha coefficient is 0.91 for the total score.

Burnout was measured using questions from the Copenhagen Burnout Inventory.²² The first section of this questionnaire measures personal burnout (fatigue, physical or emotional burnout, etc), while the second section measures work related burnout (emotionally draining work, frustration associated with work, etc), and the third section measures client related burnout (perception of the work with patients—difficult, frustrating, draining, etc). This instrument is used in several countries²² and studies have provided support for its validity. In our data, alpha coefficients were 0.88, 0.86, and 0.79 for personal burnout, work related burnout, and client related burnout respectively.

Sleeping problems were measured using five questions from the Nottingham Health Profile (NHP),²³ an instrument that allows an estimation of sleep disturbances associated with insomnia (taking medication in order to sleep, waking very early or having problems getting back to sleep, staying awake for a long period during the night, taking a long time to fall asleep, sleeping badly). The French version of the NHP was validated in a French national survey.²⁴ The alpha coefficient in our data was 0.78 for this indicator.

Data analysis

The prevalence of psychosocial factors at work and of health problems was measured. The items were grouped in indices for each of these factors. A total score was calculated for each of the indices and the distribution of scores for all caregivers was divided at the median (about 50%) in order to determine an exposed and a non-exposed group. For psychological demands and decision latitude, exposure was determined with a threshold at the median of the distribution of the general population (⩾−2 for demands and ⩽72 for latitude),¹⁵ and for social support, the threshold was the median of the distribution of nurses in a former study.²

Statistical analyses were conducted using SAS.²⁵ In all analyses, the significance level was fixed at 0.05 and a 0.10 level was mentioned as borderline. We examined two groups of dependent variables: psychosocial job factors (intermediate effects); and mental health (psychological distress, burnout, and sleeping problems) (final effects). We compared the two hospitals before the intervention on both groups of variables using logistic regression analysis. We then made two comparisons for each variable: before and after intra-group comparison and post-intervention inter-group comparison. For these analyses only care providers who responded at both interviews, in 2000 and 2002 were included.

Before and after intra-group comparisons consist of comparing the mean of dependent variables, before and after intervention (12 months after) within a sole group (the experimental or the control hospital). These analyses make it possible to track evolution over time for each of these groups. Post-intervention inter-group comparisons consist of comparing the experimental hospital to the control hospital on the post-intervention measure for each dependent variable controlling for the pre-intervention measure in each group. These analyses make it possible to evaluate intervention effects by comparing the experimental and the control hospitals.

For the intra-group comparisons, ANOVA with repeated measures were conducted.²⁶ A contrast was used to test, for each hospital, whether the mean of differences between 2002 and 2000 was null. For the inter-group comparison, ANCOVA analyses were used²⁷ to adjust the means at 2002 for means at 2000.

This research has been approved by the ethics committee of Laval University and by each ethics committee of the hospitals in the study.

RESULTS

The pre-intervention measurement (M0) was carried out by telephone interview between February and April 2000 in the experimental and control hospitals. Among eligible caregivers (n = 674 in the experimental group and n = 894 in the control group), the participation rate was 73% (n = 492) for the experimental hospital and 69% (n = 618) for the control hospital. The first post-intervention measurement (M1) was carried out in Spring 2002. Many caregivers (101/492 (21%) in the experimental hospital and (116/618 (19%) in the control hospital) had left the hospital between M0 and M1 (departure, retirement, long term sick leave, leave without pay, etc) and were no longer eligible for the post-intervention measure. The response rate at M1 among eligible subjects at baseline was 45% in the experimental hospital and 35% in the control hospital. However, the response rate with eligible subjects at M1 was 77% in the experimental hospital (n = 302/391) and 62% in the control group (n = 311/502). Table 1 shows the comparison between participants and non-participants at M1 in each hospital on their baseline characteristics (at M0). There was no significant difference between the two groups for gender, age, seniority, and job status in each hospital, or between the experimental and the control groups. However, auxiliary nurses participated more often in the experimental hospital than other caregivers.

To test a potential selection bias, we also compared the proportions of participants and non-participants at M1 on each level of psychosocial factors at work and health problems reported at M0 (table 2). There was no significant difference in the participation rate for the exposed or the unexposed to psychological demands, decision latitude, social support by supervisors and by co-workers, reward, psychological distress, sleeping problems, client, work, or personal burnout. However, in the experimental hospital, the mean of reward reported at M0 by participants at M1 was significantly higher than the mean of non-participants.

The mean scores obtained in 2000 and 2002 indicate a significant drop in psychological demands in the experimental hospital but not in the control hospital (table 3). There was also a borderline significant increase in reward (p = 0.055) and a significant decrease in effort-reward imbalance in the experimental hospital but no significant change for these two factors in the control hospital. However, decision latitude as well as support from supervisors significantly decreased in both hospitals. In summary, three psychosocial factors improved and two deteriorated in the experimental hospital while no factor improved significantly in the control hospital but two factors deteriorated.

For health indicators, the difference in mean scores between pre and post-intervention shows that sleeping problems and work related burnout significantly improved in the experimental hospital while no indicator measured deteriorated (table 3). In the control group, only sleeping problems significantly diminished while a significant increase in client related burnout and a borderline increase in personal burnout (p = 0.09) were noted.

When both hospitals were compared for mean scores obtained after the intervention, adjusting for pre-intervention scores, the mean score for each psychosocial factor was more favourable in the experimental hospital than in the control hospital. Moreover, a statistically significant difference was found between the two hospitals for all psychosocial factors measured with the exception of decision latitude, which as mentioned, significantly decreased in both hospitals (table 4).

With respect to health problems, the comparison of post-intervention mean scores between both hospitals, adjusting for pre-intervention scores, proved favourable to the experimental hospital; the mean difference was statistically significant for work related burnout and borderline for client related burnout (table 4).

DISCUSSION

Summary of results and meaning

This study found, one year after the intervention, a reduction of several adverse psychosocial factors in the experimental group whereas no such reduction was found in the control group. However, there was a significant deterioration of decision latitude and social support from supervisors in both experimental and control groups. There was also a significant reduction in sleeping problems and work related burnout in the experimental hospital, whereas only sleeping problems decreased in the control group while both client related and personal burnout increased in this hospital. The difference of means in some of the psychosocial factors between the pre-intervention and the post-intervention measures may be considered rather small, but a summation of the small differences in all targeted factors may have a clinical significance since some of the health indicators are also better in the experimental hospital after the intervention. The comparison between the experimental and control groups, after adjusting for pre-intervention measures, showed a significant difference in the means of all psychosocial factors except decision latitude. All other factors were better in the experimental group. Also, there was a significant difference between the experimental and control group for personal, client, and work related burnout, these mental health indicators being significantly lower in the experimental group than in the control group.

Psychological distress did not decrease in the experimental group and there were negative changes in supervisor support and decision latitude. This may be explained by the fact that at M1, the lag time since the start of the intervention was not very long, and most of the organisational changes recommended were not yet implemented. Since the expectations about the intervention outcomes were high among the caregivers, this may explain their reporting a deterioration in decision latitude and supervisor support and also the lack of effect of the intervention on PSI. The few changes introduced in the experimental hospital after 12 months may also explain the smaller than expected differences measured between the two hospitals in the means of both psychosocial factors and health indicators. We expect that more changes will be introduced at M2 and this should have an impact on both risk factors and health.

These results, although modest due to the short lag time between the start of the intervention and the post-intervention measure, support the effectiveness of the intervention and are consistent with other intervention research on organisational job factors and stress related emotional outcomes.^29–32 In a study among administrative employees of a United Kingdom central government department, Bond and Bunce found that increasing job control through work reorganisation allowing more discretion and choice in people’s work resulted in improvements in mental health and sickness absence rates at a one year follow up.³¹ Mikkelsen et al investigated a short term participatory intervention in healthcare institutions in Norway, and mentioned that the intervention had a positive, but limited effect on work related stress, and seemed to have started a beneficial change process.³² Kawakami et al reported decreases in depression scores and sick leave in a large electrical company intervention group, one year after organisational changes affecting psychological demands (less overtime and fewer checkpoints), decision latitude (proposition of on-the-job training and standardisation of the production process), and social support from supervisors (additional sub-leaders between supervisors and co-workers).³⁰

Strengths of the study

This research is based on sound theoretical models and the quasi-experimental design includes a control group, a large number of subjects, and several outcome measures (psychological distress, sleeping problems, and three dimensions of burnout). Moreover, the intervention targeted four well defined and theory grounded psychosocial job factors whose deleterious effects on health have been demonstrated in various work settings, and used validated instruments to measure them. This favours its generalisation outside of the hospital or the healthcare sector. Also, while means of reducing adverse psychosocial job factors may be specific to each workplace, a rigorous evaluation of the effects of preventive intervention enhances the potential for generalisation.

Limitations of the study

Our study also has methodological limits. A first limit is common to practically all preventive intervention studies on the psychosocial workplace environment. In fact, workplaces are not research laboratories and it is difficult to ensure the collaboration of perfectly comparable groups. At the beginning of the research, the healthcare organisations were continually going through restructuring and cost reducing strategies and it was therefore impossible to limit or prevent organisational changes over the study period. Thus, the use of an experimental design with randomised experimental and control groups was not feasible and the quasi-experimental design that we have chosen was the best alternative. It enabled us to choose hospitals where we were sure to have the best collaboration and where we would be allowed to follow the changes implemented during the study period which could affect the factors targeted by the intervention. A follow up of the changes happening throughout the study period in both hospitals should help establish causality links between the improvement in the psychosocial factors and the intervention and thereafter between the improvement in mental health indicators and the reduction in psychosocial factors. Nevertheless, the two participating hospitals are of the same size, offer similar acute health care, and the comparison of participants at baseline showed them to be comparable on most of the available potential confounding characteristics: gender, age, seniority, job status, work schedule, and with respect to psychosocial factors and health indicators reported at baseline (M0). In the analysis comparing the experimental and control group at M1, adjustment was nevertheless performed for each variable at baseline. However, residual confounding may have occurred and it is impossible to determine the direction of a potential bias.

A potential contamination of the intervention in the control group was limited by selection criteria for the control group: a hospital located in another part of the city and with independent management. Thus, if such a contamination occurred, it would entail an underestimation of the real difference between changes in psychosocial job factors and health indicators in the experimental group compared to changes in the control group and therefore could not explain the significant improvements observed in the experimental hospital.

A selection bias may have occurred if participants in the telephone survey at baseline were not representative of all eligible subjects. The description of participants at baseline showed their comparability to non-participants according to several characteristics available. Participation was greater among auxiliary nurses in the experimental hospital and even if their proportion among caregivers is quite small in this hospital, this could have entailed either a small over- or underestimation of the prevalence of either health problems or psychosocial work factors. Another potential selection bias could have been introduced by drop outs at the post-intervention measure or if participation at M1 was linked to changes in individual level of work psychosocial factors which are in turn linked with mental health problem prevalence. However, a comparison between those who remained in the study at M1 and those who dropped out showed no significant difference in most of the psychosocial factors at work and health indicators measured before the intervention, thus limiting the possibility of a selection bias by attrition. The greater participation of caregivers who had reported high reward in the experimental hospital may have introduced a bias in the report of this indicator and the direction of this bias is unknown. Nevertheless, we achieved a good participation rate, therefore minimising the possibility and the magnitude of a selection bias, even if it cannot be completely ruled out.

A possible Hawthorne effect (HE) may have caused an information bias as employees in the experimental group knew they were part of an intervention, the goal of which was to reduce adverse psychosocial factors at work and their effects on health. This type of bias which is susceptible to happen when the post-intervention measure takes place soon after the intervention is less likely to be present more than nine months after the intervention team’s report. However, a long term evaluation is also planned at 36 months after the intervention.

In addition, an information bias could have occurred since the work related variables were actually based on self-reported rather than objective measures. However, no objective measures were available, and the perceived psychosocial factors are possibly more important in the development of mental health problems than objective factors that may not be perceived.³³ In addition, Semmer et al reported that in studies which address this methodological issue, self-reports have been found to be “better than is often assumed”.³⁴

Conclusion

This study shows promising results regarding the effectiveness of the intervention even though only 12 months have passed since the beginning of the intervention. There was a significant decrease in several adverse psychosocial job factors in the experimental hospital, while in the control hospital, no significant improvement was noted. Furthermore, there was a significant decrease in sleeping problems and work related burnout among care providers in the experimental hospital, whereas a deterioration of client related burnout and personal burnout was observed in the control group. In light of these results, we believe that continuing the participative process in the experimental hospital will foster the achievement of a more important reduction of adverse psychosocial factors at work. It is expected that the intensity of the intervention will be directly related to its beneficial effects. Long term effects will, however depend on the willingness of management and of staff to appropriate the process of identifying what contributes to adverse psychosocial factors at work and to adopt means to reduce them. This will be assessed by a second post-intervention measure, 36 months after the start of the intervention.