You read the books and numerous articles about ROI, attended countless conference sessions, talked with “expert” consultants and even tried to implement your own with minimal results. You studied the formulas associated with ROI and have the structure in place to capture the necessary data. With so many proponents shouting the praises of training ROI, how could you go wrong, especially with this project?
This training project will teach the manufacturing employees the new work processes to be implemented when the new equipment arrives. This particular plant scores lower than the industry average for production and quality. The new processes, along with modest improvements in key machinery, are designed to boost production and quality. Your training will be the key to its success. Once you illustrate your 1,000 percent ROI (as the ROI proponents love to talk about), your training department will finally get the recognition that it deserves.
You work hard on this project. You know the exact learning objects you need to build and deliver, and you develop a low-cost, optimal blended-learning approach that will reach everyone in their preferred way of learning.
The big day comes, and your pilot case is a success. Nearly 1,000 workers participate in your final training program with excellent pre-and post-tests results. The classes were full, employees were engaged in the training, and “smile sheets” came back with the highest ratings. After the training, the workers were enthused, energized and anxious to apply their new skills. You kept your costs to only $20,000—quite a bit lower than you had budgeted. Everyone complimented you and claimed that this training material was perhaps the best that your organization ever offered. Success!
Soon after the training was completed and the first training ROI calculation was completed, you hear the rumblings. The budget people challenge the necessity of purchasing the modest upgrades to the machinery. “Why invest in upgrades if we may close the plant?” they ask. The plant supervisor concurs that without the machines, the new processes will not work as designed. The highly enthused, excited and energized workforce that you delivered just days ago is now disgusted, upset and less motivated than ever before. Production drops, as does the quality of work. Rumors of the plant closing affect everyone. Production drops further. Quality is a joke. The plant loses nearly a million dollars. The project is a disaster.
You calculate your ROI. The project lost $980,000, plus the $20,000 spent on training. Calculating your training ROI yields -500 percent ROI. (-$980,000 – $20,000)/$20,000 = -500 percent ROI) You would have had better results if you’d spent a million dollars instead. Your heart sinks deep into your stomach. What went wrong?
The next phone call you get will be your boss.
Training ROI refers to “training return on investment.” It is a hot topic on the speaking circuits at many of the training conferences.
In the preceding case, did the training fail? Did the organization fail? Should training be penalized even though it presented a superior product? As the case illustrates, training did not fail, but the metric used to measure training sure did.
The biggest value that training ROI ever produced was to sell books for a few authors, so unless you are one of those authors, forget all you have heard and read about it. Training ROI is perhaps the least meaningful metric that you can determine when it comes to training, rated just below “smile” sheets. It is not indicative of the contribution that training makes to an organization.
The claims of training ROI proponents are impressive. Training ROI is 100 percent they shout! Two hundred percent! Even 1,000 percent! Do not fall for the hype; and certainly do not stake the future of your training department on it.
Others have interchanged the word “return” with “revenue” or “investment” with “expense.” Those interchanges are invalid.
Nonprofit
Consider this scenario. A product line is struggling with sales, and the brand manager asks you to put together sales training for her sales team. You accept the request and put your trademarked vigor into the process. The year before the training, the sales team was selling 1 million products at $10 each. The year after running everyone through the sales training, they were selling 2 million products at $10 each. Sales doubled. The sales training worked and did what it was asked to do.
If a $20,000 sales training class produced an extra $10 million in sales, what would the ROI proponents claim the training ROI is? Don’t bother to do the math; it is inconsequential. Before you start drooling at the success of your training ROI, be sure to factor in that the company’s cost to produce and sell the product is $12 per unit. The extra million units sold, while generating $10 million in additional revenue, ended up costing $12 million in production and sales cost, for a net loss of $2 million. Ouch. Another great job by the training staff produces yet another negative training ROI.
The biggest problem with the ROI claim is the letter I, which stands for “Investment.” Contrary to almost all thinking in this profession, training is not an investment—it is simply an expense. While “investment” sounds more important than “expense,” training is nonetheless, an expense. “Investment” is a business term that implies the adding of capital to an organization. Unfortunately, “human capital” does not qualify. Until a training vendor will sell a course to you, then buy it back for more than you paid, you did not invest in training, you spent money on training. Peter Drucker was right when he said, “Marketing and innovation make money. Everything else is a cost.”
It is not that expenses are bad—they are not. Businesses cannot operate without spending money. What the business leaders want to know is whether the money was spent wisely.
Before you object to the strict definition of “investment” as a matter of semantics, let us clear the air that other popular terms such as “payback,” “break-even” and “cost-benefit,” while more meaningful than “ROI,” are also of dubious value.
Another common phrase expressed with training ROI is, “Training Drives Business Results.” Training no more drives business results than a car’s steering wheel drives a car. Vice presidents drive business results because they have the power to assemble resources that affect change, including engineering, marketing, legal, production and, naturally, training.
Consider a jazz band. People swoon to the sound of the mellow tunes belted out by the saxophone player. They hardly pay attention to the drummer. The drummer is context, while the saxophone player is content. If the band were to do an ROI analysis on each instrument, however, perhaps they would eliminate the high expense of the drummer and “invest” in two more saxophone players. How much more successful would the band be with three saxophone players and no drummer? Not much actually, because while context does not get the marquee status, there is no marquee without it.
My college roommate worked at a local deli, one of several in the small town catering to the college students. With so much competition, the delis were price-competitive on their subs and sandwiches, almost to the point of selling them at break-even prices. My roommate told me that the owner didn’t care because he made all of his profit on the soft drinks, tea and coffee. His customers knew the prices of his sandwiches and his competitors’ sandwiches, but never mentioned the price of the drinks. If the owner had done an ROI analysis on his business, he would have eliminated everything but the drinks, and he would have found himself quickly out of business.
The concept of measuring any single department for its ROI is offensive to the team concept of an organization. With that mindset, why would the organization not just keep the top two or three departments and let all the rest go? “Sorry production, marketing and accounting, but your ROI did not meet our requirements so we are letting all of you go. Instead we are putting all of our money in legal, distribution and training.”
If You Insist
In the “If you insist” department, you are determined to prove the effect training alone has on the bottom line. Actually, there is only one way to do this, and unfortunately, it is flawed. The test is known as the “A-B test” and is often used in marketing and science. In an A-B test, all factors are kept the same except for one. Tests are run, half with the factor set one way (A) and half set the other way (B). The results are then compared to define the effect that the factor had on the outcome.
For example, a marketing campaign will send out identical messages, except half will have one offer, for example, and the other half will have a different offer. They will then compare the results to see what effect the offer itself had on the bottom line.
To do the same in training, you would need two identical projects. Both projects would have to be run the same way: equipment, product, process, time, abilities of the staffs, etc., except that in one project you would add training and in the other project you would not add training. Understand that this would be difficult to set up. How likely are you to find two identical projects? How likely is your organization to allow such a test? What purpose would it serve—just to prove training works? Why not conduct similar tests for legal, accounting and shipping? When completed, the test would only prove the effect training had on one project. It would be a stretch to make an inference about all training from a sample of one. If you ever have such an opportunity, it would be interesting to run the A-B test on training, but interest would be its only value. Making serious decisions based on one such test would be risky.
Training Metrics
If training ROI and A-B testing are of dubious value, what metrics are available that accurately measure training?
Successful training organizations deliver effective and appropriate training that is applicable to the organization’s needs in an efficient manner. Instead of chasing the elusive and meaningless training ROI, other training metrics are better suited to demonstrate training’s impact. None of these by themselves are necessarily meaningful, but used collectively they are eye-opening and are well suited to illustrate training’s value. No training metric will overcome an organization that fails to value training’s role in achieving business objectives.
Effectiveness: The effectiveness of training is a measurement of learning. It is determined by comparing post-test scores with pre-test scores and then measuring the net change. There are several methods to measure this—on a per-student basis, on a per-“skill point” base or on a per-dollar basis. Let’s look at the skill-point base, which measures the cost of raising a student’s skill by one unit.
Effectiveness Skill Cost: To measure the cost of adding a skill unit, determine the net change in skill ratings by doing pre- and post-testing. If the net change averages 10 points and you have 500 students take your training course, you have a net improvement of 5,000 rating points. If your total cost for the training was $20,000, your Effectiveness Skill Cost was $20,000/5,000 = $4/skill unit (point). If a training course provides multiple skills, prorate the cost of the training to all skills, or your costs will be skewed high. After some time spent tracking these values, you will have benchmarks of your Effectiveness Cost for various courses, skills, media and perhaps departments.
The lower your Effectiveness Skill Cost, the better your training is performing. To lower your Effectiveness Skill Cost, either work on driving up post-test scores in your training or work on driving down the cost of delivering the training.
Effectiveness Outcome Cost: Often, the goal of a training event is not to drive up skill-rating points for all students. Rather, it is to train the students to perform at a certain level. If that level, for example, is a rating of 80, training that moved a student from a 40 to a 70, while admirable, does not meet the objective. Instead, you want to know how effective your training was in having students leave with a rating of 80 or higher and what the cost of that training is per student.
Again, we use pre- and post-testing to capture the results. After pre-testing, count the number of students who have already met the required score. After post-testing, count the students again who have met the required score. You are interested in the difference, the reason being that if the student already met the required rating, this training event cannot take credit for that rating. Divide the cost of the training by the difference. Using the same example as above, 500 students go through your training, but pre-testing shows that 175 already have the required rating. After the training, post-testing shows that 300 now have the required rating, for a net gain of 125 students. Dividing the cost of the training ($20,000) by the net gain (125) gives an Effectiveness Outcome Cost of $160, meaning that it cost approximately $160 to have another employee perform that skill at that level.
The lower the Effectiveness Outcome Cost, the better your training is performing. Here you have more room to maximize your training’s effectiveness than you did with the Effectiveness Skill Cost value. First, invest in having the pre-test done before the employees enroll in the training, then cancel those students (where possible) who already meet the requirements. (It could be easily defended that ensuring a skilled workforce is a higher objective than the process of developing the workforce, therefore it is training’s goal to eliminate unnecessary training.) Second, if feasible, cancel the students who likely will not pass, or set up a prerequisite course so that they have a better chance of succeeding. Both of these methods will help reduce your costs.
Efficiency: The efficiency of training shows how much training was delivered per some unit, for example, the number of employee training hours per $1,000. A full classroom is more efficient than one that is half-full. If a 10-hour class cost $1,000 to run, its efficiency level for 10 students in the class is 100 training hours (10 students times 10 hours)/$1,000. The same class with 20 students produces 200 training hours/$1,000, and thus is more efficient.
The metric also applies to e-learning. A one-hour e-learning object cost $50,000 to produce. If 5,000 students take the e-learning object, the efficiency is 100/$1,000 (or 5,000/$50,000).
Higher efficiency ratios indicate more efficiency than do lower efficiency ratios. To improve your efficiency ratios, focus on filling your classrooms for instructor-led training and on delivering your e-learning content to more employees.
Notice that the effectiveness metrics focus on quality, and the Efficiency metrics focus on quantity. Your training is moving in the right direction if your effectiveness costs are going down and your efficiency ratios are going up.
Effectiveness and efficiency are noble and honorable metrics, but they ignore the demand for training and the business reasons for doing the training. The next two metrics address those concerns.
Applicability: The applicability ratio shows how the training is aligned with the business objectives. What good is raising employee skill ratings and putting more employees through training if the organization does not need those skills? The applicability ratio is a simple calculation, but it is rather difficult to gather the supporting data.
To calculate the applicability ratio, divide those training costs that are identified for specific skills in “demand” by the organization by the total for all training costs. This can be done for a particular course or for the entire training operation.
For example, five courses are offered, of which three identify specific skills that the organization needs to meet its business objectives. The cost of each student is $1,000. Sixty employees take the courses tied to business objectives, and 20 employees take the other two courses. The applicability ratio was (60 x $1,000)/(60 + 20) x $1,000) or 75 percent, meaning that 75 percent of the training costs apply to specific business objectives.
The value of the applicability ratio is that, implemented correctly, it promotes the integration of training for specific business objectives by adding organizational rigor to the training process.
Appropriateness: The appropriateness ratio shows that the right learning objects are delivered to the right people. Having excellent training is wasted if you are teaching computer programmers how to sell, or if you are teaching social workers how to program computers. It is similar to the applicability ratio, but provides more accountability by measuring the training of the specific people assigned to specific business objectives. Such training can be linked by job function, project, team or business objective.
To calculate the appropriateness ratio, divide the training costs for training those people identified with specific business objectives by the total cost of all training.
In the applicability ratio example above, 24 of the 60 people enrolled in the course-specific classes are identified as being there for a specific business objective, so we divide the cost of training those 24 people by our total training costs. Thus the appropriateness ratio is (24 x $1,000)/(80 x $1,000) = 30 percent, which says that 30 percent of our training costs were spent training certain people for specific skills needed to meet our business objectives.
The appropriateness ratio promotes training for specific business objectives while it discourages superfluous training—imagine being able to give priority to students who need the training for specific business objectives.
If you were presenting an annual report of training, would you rather say that your training this year compared to the previous year was:
- 20 percent more effective.
- 12 percent more efficient.
- Provided 25 percent more skills valued by the organization.
- 30 percent tied to training specific people for specific projects.
Or would you rather say that your training ROI was negative 500 percent?
Would you rather be judged by your negative ROI? Or would you rather talk about the effectiveness, efficiency, applicability and appropriateness of your training? If you were fighting for valuable budget dollars, which metric would you use?
Bob Dust is the founder and general manager of Gyrus Systems, a company with 23 years of experience developing software products for the training industry. He can be reached at bdust@gyrus.com. Visit Gyrus Systems’ Web site at www.gyrus.com. Bob is also an instructor in the School of Business at Reynolds Community College jsr.vccs.edu.
