Pouring money into a theory is risky, especially when there is no documented evidence to support that theory.
An associate of mine who works for the Department of Defense was in search of evidence that proved simulations, games and related training technologies improved performance in live situations.
He suggested to his superiors — both civilians and military officers — that they spend precious research and training dollars on what appeared to them to be unproven ideas. The fact that these “unproven ideas” seemed (to my acquaintance) quite obviously to work, simply wasn’t good enough.
As the co-founder of a firm that focuses on simulation-based training and serious games, I was sure hard facts existed regarding the effectiveness of using computer-based simulations for training.
I began to survey literature in preparing to write this article, and what surprised me the most was not that I found exactly the kinds of research for which I was looking but, rather, the breadth and sheer volume of such research — in every industry I examined, I found that people already had studied the use of simulations for training. In the reports I read, the results ranged from guardedly optimistic to wildly enthusiastic.
Having conducted the literature survey for this article, my belief is that over the last few years, as simulation-based training has made the transition from high-cost, dedicated hardware systems to inexpensive, PC-based, virtual training solutions, and as a new generation of developers and managers has entered the industry (often with backgrounds in video game development), we have been unaware of the considerable research already in existence that documents the effectiveness of what we were building.
This is not a comprehensive survey of the literature on the effectiveness of simulation-based training — such a survey of only one industry, such as medicine, would have to cover dozens upon dozens of citations. This article is instead a quick overview of a dozen or so citations that demonstrate the breadth of research in this area, as well as its surprisingly long history.
Pilot and Aircrew Operations
In 1992, the Naval Training Systems Center (now The Naval Air Warfare Center Training Systems Division) conducted a meta-analysis of 247 articles, research reports and technical reports, studying the effectiveness of simulation for pilot training.
Of these, 26 experiments were identified as having sufficient information for statistical meta-analysis, and 19 of these experiments focused on jet pilot training. Among jet pilot training studies, more than 90 percent of the experimental comparisons favored simulator and aircraft training over aircraft training alone.
In a study of the effectiveness of virtual training for crew resource management (CRM) conducted with the U.S. Air Force 314th Airlift Wing, a low-cost, PC-based simulator using Microsoft Flight Simulator was designed to elicit the communication and crew coordination behaviors associated with instrument and visual airdrop missions. Treatment group students received a four-hour training profile before their first airdrop flight, and control group students did not. CRM performance ratings during the first subsequent airdrop flight were significantly higher for treatment group students than for control group peers.
Higher performance grades in training records also were observed for treatment group students in all CRM skill areas through subsequent flights, with fewer sorties to meet criterion for communication, crew coordination, task management and decision making for both navigators and co-pilots. For the two targeted skill areas of communication and coordination, simulator-trained student navigators required, on average, 10.2 sorties to achieve proficiency, compared with 11.8 sorties for their control group counterparts — a reduction of about 13.6 percent.
Medical Specialties
In 1969, a study conducted at the School of Medicine of the University of Southern California evaluated the use of a physical patient simulator, Sim One, in training anesthesiology residents.
The study demonstrated that simulator-trained anesthesiology residents required a mean of 9.6 endotracheal intubations to achieve a skill level high enough to perform four consecutive professionally acceptable intubations, compared with 18.6 intubations for residents not trained on the simulator to reach the same standard.
The same study showed that simulator-trained residents achieved the most exacting evaluation criterion applied in 55 days of training, compared with 77 days for their nonsimulator-trained counterparts — a savings of 22 days, or more than 28 percent.
More recently, a randomized, double-blind study at the Yale University School of Medicine’s Department of Surgery evaluated residents trained in laparoscopy using a virtual reality (VR) system, Minimally Invasive Surgical Trainer-Virtual Reality, from Swedish firm Mentice AB.
The study found that VR-trained residents performed significantly better during cholecystectomy (gall bladder removal) surgery than a control group not trained using the VR system — VR-trained residents worked 29 percent faster than control group participants, and errors were six times less likely to occur during their surgeries. Control group participants were five times more likely to injure the gall bladder or burn nontarget tissue and nine times more likely to fail to make progress for a minute or longer at some point during surgery.
Equipment Maintenance
A researcher at the Institute for Defense Analyses extracted 11 studies in which simulated equipment was used to train maintenance technicians. These studies compared instruction with the simulators to instruction with actual equipment, held overall training time roughly equal and assessed final performance using actual (not simulated) equipment.
Over the 11 studies, the use of simulation suggested an improvement from 50th percentile to about 66th percentile achievement among students using simulation. The initial investment for simulated equipment averaged 43 percent of that for actual equipment, and the operating costs for simulated equipment averaged 16 percent of that for actual equipment.
At the request of the Canadian Air Force, the Canadian Forces School of Aerospace Technology and Engineering conducted a study in which it adopted a virtual training system for what had been a two-and-a-half-day training course that covered maintenance and removal of propellers for C-130 aircraft.
The virtual training system consisted of CAE Simfinity’s Virtual Maintenance Trainer and NGRAIN’s Virtual Task Trainer. After one day of study, all students passed the practical examination with an average score of 94 percent. This represented a 60 percent reduction in training time.
Commenting on the study, a Canadian Air Force officer said, “Trainees can acquire knowledge faster when 3-D equipment simulations supplement traditional methods.”
Military Combat
The U. S. Army Research Institute for the Behavioral and Social Sciences and the University of Louisville examined the instructional effectiveness of the U. S. Army National Guard’s (ARNG) Reserve Component Virtual Training Program (RCVTP) — structured exercises conducted in a simulation training environment, the Simulation Networking (SIMNET) training system — to provide ARNG armor units with intensive training experience during their weekend drills or annual training periods.
After 12 hours of training using RCVTP, units took 46.1 percent less time to complete their tasks while making 76.1 percent fewer errors and requiring 84.4 percent less coaching.
In 1991, the institute compared results from 714 platoons that received conventional training in the Armor Officer Basic Course with 39 platoons that received training based on networked simulation using SIMNET. It found that networked simulation both improved field performance ratings by 25 percent and saved 20 percent of time in the course.
Education
Researchers at Rice University conducted a study in which the effectiveness of simulations for teaching statistical concepts was compared with the effectiveness of a textbook.
Subjects trained with the simulation (developed at Rice) outperformed subjects trained with the textbook on seven out of eight critical questions, and they were more able to recognize the key elements of ill-defined problems embedded in various real-world situations and apply the relevant statistical principles.
According to the authors, “this provides support for the thesis that simulation is effective for training on educational and cognitive tasks (as opposed to tasks such as flying an airplane where simulation has been shown unequivocally to be effective).”
Law Enforcement
The Justice and Safety Center at Eastern Kentucky University studied the use of a mobile, trailer-housed firearms and judgment simulation system — Professional Range Instruction Simulator (PRISim) from Advanced Interactive Systems — for training law enforcement officers.
After two hours of using the system, officers placed an average of 31.6 percent more of their shots on target, and they were 52.9 percent less likely to fire their weapons without justification.
“The system appears to be beneficial in building and/or enhancing skills that are arguably the most important for the safety of the officer and others, i.e., accuracy, effective use of cover, avoiding the unintentional shooting or endangering of innocents and ensuring the shooting is justified,” according to the study’s authors.
Driving and Trucking
Long-haul trucking firms have been adopting simulators for training and are quantifying the results, according to an article in Heavy Duty Trucking magazine. One firm, Schneider National, saw a 21 percent reduction in preventable accidents in drivers’ first 90 days after simulation-based training.
“One hour of training in the simulator is equivalent to three or four hours of training over the road,” said Don Osterberg, vice president of driver training and safety.
Another trucking firm, Bison Transport, saw its accumulated safe-driving miles increase by 50 percent after adopting simulators for training.
“We’ve observed an 83 percent improvement in mean time between incidents after simulator training for preventable accidents,” CEO Don Streuber said.
Both long-haul trucking firms use the Mark III and TranSim VS simulators from MPRI.
Additionally, the University of Utah conducted a study of a pilot training program at the state Department of Transportation that was developed for snowplow operators using the Mark II and TranSim VS driving simulators.
According to the authors, in the six months following training, “The odds of getting in an accident were lower for the group of drivers who received training compared with a matched control group who did not receive it.”
They also noted a 6.2 percent improvement in fuel efficiency for drivers receiving the simulation-based training.
Conclusions
In every survey and every industry examined, simulation-based training was seen to have a positive effect.
In every case in which simulation-based training was directly compared with traditional methods, simulations were observed to be superior on some or all criteria.
In every case where the costs of simulation-based and traditional training were compared, simulations were found to be less expensive, whether because of lower acquisition costs, lower operating costs or lower costs resulting from more effective or faster training.
In every case in which the authors of a study made a recommendation about the use of simulation-based training, they recommended its ongoing or expanded use.
In short, simulation training has demonstrably reached the point where questions of its fundamental effectiveness should no longer play a part in evaluating its potential use for any given project.
In conducting such evaluations, the basic usefulness of simulation training now can be taken as a given, allowing project planners and decision makers to focus their attention on their specific applications and how best to use simulation training in the most useful and cost-effective manner.
Frank Boosman is the chief operating officer of 3Dsolve Inc. He can be reached at editor@clomedia.com.
