US corporations spend $55 to $60 billion per year delivering safety training programmes in the workplace to an estimated 60 million employees,¹ but the effectiveness of many of these programmes, especially computer programs, has not been well studied. A 1998 literature review on health and safety training by Cohen et al noted that 80 out of 2000 articles met the standard definition for evidence of effectiveness.² The effectiveness of various types of interventions may have a direct effect on the safety of workers, and the National Institute for Occupational Safety and Health (NIOSH) has identified intervention effectiveness as one of its priorities in the National Occupational Research Agenda (NORA).³ The goal of intervention effectiveness research is to show the impact of interventions, such as education, in the prevention of workplace illness and injury. Organisations have been increasingly turning to computer based training as a cost effective means of delivering training programmes, but the study of various instructional design methods to maximise learning and transfer has not kept pace with the development of this technology,⁴ leaving instructional designers with little theory based evidence to guide their designs. Although there is a need to understand how adults learn from computer delivered training, there have been few investigations of multimedia learning theories using adults as learners.

Designers of computer based training have many options, including various types of presentations and interactions. This study was undertaken to determine high and low level learning outcomes for adults after receiving various types of multimedia based training presentations.

Occupationally related lung disease is a significant problem in the United States. Respiratory illness accounted for 7.5% of all non-fatal occupational illnesses in 2002.⁵ Among all pneumoconioses there were more than 31 000 deaths in the USA from 1990 to 1999, accounting for more than 300 000 years of potential life lost, and asbestosis deaths have increased from fewer than 100 in 1968 to over 1250 by 1999.⁶ Occupational asthma is increasing worldwide, especially in the industrial countries, and it is estimated that 10–15% of adult asthma cases are related to occupational exposures.^7–9 Occupational asthma affects as many as 400 000 to 3 million workers in the United States every year.¹⁰ By 1990, occupational asthma had become the most compensated occupationally related lung disease in Canada.¹¹ The major interventions for occupational lung diseases are primary prevention through good industrial hygiene practices, engineering controls, respiratory protection, and worker education. An estimated 5 million US workers are required to wear respirator protection to prevent workplace exposures and the resultant diseases.¹²

Companies in the United States are mandated by federal law, the Occupational Safety and Health Act (OSHA Respiratory Protection Standard 29 CFR 1910.139),¹³ to provide yearly training for those workers who may need to use respirators in the workplace.

Health and safety training is increasingly being delivered by computer, either on CDs, over company intranets, or over the internet.¹⁴ Well designed computer based training can be cost effective,¹⁵ and can be an effective training tool.^16–18 However, despite this increase in the use of computers to deliver training, less is known about the effectiveness of various computer based training methods than is known about other forms of training.^4,19 The lack of empirically based guidelines has left training designers with insufficient direction for designing online training. The choice of various methods of presenting information is important for safety training since the consequence of being poorly trained can result in injury or illness. Under OSHA regulations every person who uses a respirator is required to receive training, but research has shown that simply making sure people hear or see a training presentation does not mean that they will be able to use the skills or knowledge that are covered in training. Increasing the interactivity of computer based training may be one way to increase its effectiveness. Eckerman et al showed in a 2002 study that interactive computer based training was significantly better than a computer program that was non-interactive when teaching respiratory protection.¹²

One of the central paradoxes of all types of training and education is the problem of transfer.^4,20 Transfer refers, in part, to the ability of learners to make use of their newly acquired knowledge outside the classroom or learning situation. The problem educators and trainers face is that far too often, learners who appear to have acquired knowledge and skill and can answer questions in the classroom, are unable to apply that knowledge and skill in work settings.

There is a need to understand how various methods of computer based training work to enhance comprehension and transfer. Cognitive theories of learning²¹ offer important guidelines for the development of effective multimedia based training. The purpose of this study was to examine learning after various methods of computer based training that included instructional text only, text with animation and pictures, or narration with animation and pictures.

The first hypothesis is that instructional multimedia that utilises both visuals and narration will produce better comprehension of the training material than will training that does not utilise narration. The second hypothesis is that instructional pictures and animations will result in better comprehension than instruction that does not utilise these tools.

METHODS

Participants in the study were 81 workers (57 women and 24 men) at a medium sized manufacturing plant in a large US Southwestern city. Participants were recruited through a sign-up sheet on a bulletin board at their place of work, and the study took place in a computer lab at the plant. Participants were paid $50 for their participation in the study. After signing a consent form, subjects completed a demographic survey which included questions about age, preferred language of instruction (English, Spanish, Navajo, or other), highest grade completed in school, job title, and how often computers were used at work or home. Participants were then pre-tested on their understanding of respirator concepts and given an English literacy test, the S-TOFHLA.²² The participants were randomly assigned to one of the three computer based instruction groups. Thirty participants who displayed an understanding of basic respirator concepts because of previous respirator use training, who had more than 14 years of education, who indicated that their primary language was not English, or who had technical or computer problems during the training were excluded from the analysis.

A 2645 word instructional text was developed. The instruction was designed to cover the material required in OSHA Respiratory Protection Standard Regulation 29 CFR 1910.139. The text contained information such as what the employer’s and employee’s responsibilities are, how to clean and maintain respirators, how to store respirators, and how filters and cartridges work. The text was written to have a high degree of coherence²³ and to be easily understandable to readers with a high school degree. The Flesch-Kincaid Grade level score was 8.5 as measured by Microsoft Word.²⁴ This text was developed into three instructional programs using Macromedia Director.²⁵

One version of the program consisted of text information only (TXT), one consisted of text with animation and pictures (TAP), and one consisted of text, narration, animation, and pictures (NAP) (fig 1). Participants were randomly assigned to one of the three versions of instruction.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

 Greyscale images of screen shots of the various programs. A screen describing an air supplied respirator from version TAP (A) and version TXT (B). A screen with an animation from version NAP (C) and the same description from version TXT (D).

The first version, TXT, presented the text in a 43 page program (average of 61 words per page). The second version, TAP, was identical except that it also contained instructional pictures and animations. Animations were used to deliver instruction on how filters and cartridges work and how to perform positive and negative fit tests (fig 2) and were designed to foster the learner’s construction of mental models of various concepts of respirator functioning such as how filter and cartridges work, how they can fail, and what happens if they fail. On the pages with text and animation the animation appeared above the text. The animations were designed to be short and to cover a single concept so as not to overload learner’s working memory capacities.²⁶ Ten animations were developed. The average length of the animations was 15 seconds. Pages without animations had instructional pictures that were intended to help learners understand the material in the text.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

 Greyscale screen shots of some of the animations used in version NAP showing chemicals sticking to a cartridge (A), chemicals passing through a cartridge that was not designed to stop them (B), a filter trapping particles (C), and a positive fit test (D).

The NAP version was the same as TAP except that the text on the screen was narrated.

Immediately after completing the instruction, participants were given two tests in succession: a multiple choice test that tested their knowledge of material that appeared word-for-word in the instruction; and a transfer test that tested their ability to use their knowledge to solve problems in respirator use and in respirator health and safety issues encountered on the job.

The multiple choice test was designed to test a low level of learning at Bloom’s²⁷ knowledge level for recall of information; it consisted of 12 items which repeated the information in the text. For instance, the text had the sentence: “Particle filters can only stop particles like those produced by sanding or grinding. They cannot stop gases and vapors, like those produced by painting and welding. Gases and vapors will pass right through a particle filter. A filter will only stop particles.” One of the multiple choice questions asked:

• Particle filters can stop

  1. gases and vapors as well as particles

  2. anything that you might breathe in

  3. gases and vapors

  4. only particles.

The transfer test was designed to test learning at the higher levels of analysis, synthesis, and evaluation²⁷ and the subject’s ability to use information in new and creative ways. The ability to use information in new and creative ways is a sign of transfer.^20,28 The transfer test consisted of five items. Subjects were asked to write down all answers they could think of for each of the questions. For instance, one question asked: “You and a co-worker are in a room with painters. You are both wearing respirators. Your co-worker says that he smells solvent. Write down everything that could be wrong.” Correct answers included statements that the co-worker may not have a clean shaven face and the respirator not fitted correctly, that the cartridge may be full, and that the respirator may not have been positively and negatively fit tested before use. This test was scored by counting the total number of correct answers given compared to a list of all possible correct answers.

RESULTS

Fifty one participants were included in the analysis. The ages of the participants ranged from 26 to 64 (mean 42.8, standard deviation 10.8). The average grade completed as reported by the participants was 11.8 with a standard deviation of 1.0. Sixty one per cent reported having computers at home. Eighty four per cent reported that they used a computer at work at least once a week. Forty seven per cent reported that they used a computer every day for work. There were no significant differences between the groups in age, computer experience, prior respirator knowledge, or years of education. Gender was not recorded in a way that it could be matched to individual scores, so its effect is unknown.

An ANOVA calculated on the results of the multiple choice test, which measured lower level learning, indicated that there were no significant differences between the groups: TXT (mean = 10.7, SD = 1.3), TAP (mean = 10.4, SD = 1.7), and NAP (mean = 11.0, SD = 1.2) (fig 3), F(2,46) = 0.778 (p = 0.465).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

 Mean scores and standard errors of measurement on the multiple choice test (A) and transfer test (B).

On the transfer test, which measured higher level learning, there were significant differences between the treatments, ANOVA F(2,46) = 11.951 (p<0.001). The means and standard deviations for the groups on this test were TXT (mean = 6.6, SD = 2.5), TAP (mean = 10.5, SD = 4.1), and NAP (mean = 13.9, SD = 5.4). A Fisher’s PLSD post-hoc test revealed that the differences between each of the three groups was significant at the p = 0.05 level.

DISCUSSION

Computer based training is increasingly a tool of choice for training. Despite the many options available to designers, many computer based training modules are text based.¹⁴ This study examined learning after various methods of computer based training that included the common form of instructional text only compared with text with animation and pictures and narration with animation and pictures. The results of this study suggest that computer based training designs, based on cognitive psychological principles, that have been shown to be effective for younger learners, may also be an effective method for training adults. However, these results have to be considered in the light of the small sample size of the study and the possibility that the results have been influenced by chance or factors that were not controlled for. The study also suggests that methods based on cognitive psychological principles of multimedia development may be more effective than text for instructing adults for high level learning goals in occupational safety and health. High level learning outcome is believed to be important because it is associated with transfer—that is, the ability to use new information in a new way.²¹

Transfer is one of the most basic learning goals, and educators and trainers have spent much time and effort on developing methods to increase it among trainees.^4,20,28,29 Trainers can easily underestimate how well learners will be able to use knowledge gained in training on the job. This may be a significant problem for computer based training where there is a greater disconnect between the training designer and the trainee. Historically, there have been three approaches to increasing transfer: general transfer, specific transfer, and specific transfer of general knowledge.²⁰ General transfer refers to an approach that tries to improve a subject’s general ability to learn. Specific transfer refers to the idea that training should teach skills exactly as they are used in the real world. Specific transfer philosophy has guided many training projects;⁴ however, the complexity of “real” situations make it difficult to replicate in a training environment and this type of training has not been very successful. The middle ground between these two approaches is thought to be the preferred way to ensure transfer of skills.^4,20,28 If learners are taught basic concepts that help them develop mental models of how systems function, they are more likely to use the new information in “real world” settings. Conceptual understandings, or mental models,^21,30 of systems are knowledge structures that contain specific information that can be used to accomplish various tasks but are also flexible enough to used in complex “real world” settings.

Mental models of system functions can be developed in learners with multimedia designed to support learner’s cognitive functions.³⁰ Knowledge can be structured and stored in many different ways. Simple knowledge structures may not consist of more than one-to-one correspondences between facts. For example, a trainee may know that “a black cartridge is for organic vapours” but not understand what a vapour is, what organic means, or where they might find organic vapours. The information is useful for answering a question that asks: “what colour cartridge do you use for organic vapours” but by itself, is not useful for further problem solving. Mental models are more complex knowledge structures where the key points of a system are known along with their relationships and the interrelated influences on each other. For instance a well developed mental model of an organic cartridge will contain information about how cartridges work to stop vapours, how the cartridge can fail, and what vapours are and where they are located at the job site.

It is interesting to note that our results confirm earlier studies which have concluded that different types of instruction on the same topic can lead to different levels of understanding.³¹ This study found no significant differences between subjects on the multiple choice recall test, suggesting that each of the treatments was equally effective in generating simple, one-to-one connections of information. On the higher level recall test, however, participants who received information in both narration and text and pictures were better able to answer questions asking for unique and creative answers, suggesting that their knowledge structures were more complex and robust.

Computers can be powerful tools for the development of mental models because of their ability to deliver multiple types of information in different modes such as pictures, sound, animations, movies, and text. Because so many different combinations of information are possible, designers need empirically derived guidelines to help them determine which mode of information would be best in a particular learning situation and to avoid adding information that might seem entertaining or motivating but might result in lower levels of learning.³²

Much of the theoretical development of multimedia learning theory has been carried out in experiments using relatively young, well educated college students as the subjects.^32–35 Few studies have looked at multimedia learning for adults of different abilities, ages, or education. The few studies that have been done suggest that when using computers, adults may learn differently than their younger peers.^36,37

In a study of younger learners, information presented in both visual and verbal formats was found to more easily remembered than information presented in either a visual or a verbal format alone.³⁸ This and other observations has lead to the Generative Theory of Multimedia Learning.^21,30 The Generative Theory of Multimedia Learning describes an information processing system that has both verbal and non-verbal components. The verbal components process language and the non-verbal component processes other types of information such as pictures. Information that is presented in both visual and verbal forms can be more efficiently processed than information that is focused more on one or the other of the processing systems. Information that is presented in both a visual and a verbal format may be easier to remember since connections are made between the visual and verbal information, making either a retrieval cue.

Both verbal and non-verbal systems can become overloaded; designers must carefully consider how information is presented to learners. It is important to structure how information is presented to novices.³⁹ Narration with pictures or animation utilises both the verbal and the non-verbal processing capacities. Text with pictures and narration would be expected to result in lower levels of learning because the non-verbal system has to be used to both examine the pictures, and to read the words on the screen.

The results of this study are consistent with the predictions from Generative Theory of Multimedia Learning. Participants who received narration with pictures and animation (NAP) scored higher on a transfer test than did learners who received instruction either with text and animation and pictures (TAP) or who received instruction with text alone (TXT).

The results of this study offer important points for designers of computer based training. The results were consistent with previous studies showing that learners receiving instruction as narration with pictures and animations were able to answer transfer questions significantly better than learners who received instruction in the other modes. This result suggests that learners receiving instruction in both visual and verbal formats had better developed mental models of how respirators work and might be better able to transfer this information to “real” work situations. Designers of computer based training in occupational safety and health should consider presenting multimedia materials with coherent visual and verbal elements.