Health care company Kaiser Permanente needed to come up with a more standardized, cost-effective approach to learning and follow-through across the enterprise. How did they do it? By thinking small.
If the viability and sustainability of today’s training organization rest on its ability to deliver results in terms of Kirkpatrick levels 3 and 4, plus Phillip’s return on investment (ROI) — sometimes referred to as a Level 5 evaluation — how does a company with more than 160,000 employees and 300 training departments adopt a common approach, ensure learning transfer and gather meaningful data to sustain and enhance training programs?
Those were the questions Robert H. Sachs, Ph.D., vice president of learning and development at Kaiser Permanente (KP), pondered throughout 2007. Working in conjunction with Kaiser Permanente’s National Learning Leaders (NLL), the group concluded that two things were needed: a common terminology and approach across the large number of decentralized learning organizations and a scalable method for managing the post-program follow-through period.
Creating a Common Language and Approach
For a common language and a consistent approach to learning development, KP adopted The Six Disciplines of Breakthrough Learning by authors Calhoun “Cal” Wick, Roy Pollock, Andrew Jefferson and Richard Flanagan. The book’s focus is the six disciplines that may be applied to increase the impact of any learning and development program:
1. Define business outcomes.
2. Design the complete experience.
3. Deliver for application.
4. Drive follow-through.
5. Deploy active support.
6. Document results.
Since 2006, NLL has been leading the effort to embed these six disciplines to increase learning’s impact on business results. Decentralization of the approach throughout KP is the key factor for widespread adoption of the Six Disciplines as a common learning solution. NLL adopted them through the process of reading and discussing the book, inviting Wick to KP to speak with NLL, creating tools and templates for all regional learning councils to use throughout KP and holding internal training sessions.
Alan Jang, senior manager of the learning solutions team, said, “In order for learning solutions in KP to be successful, meaning achieving measurable results that impact the business, we had to create a new finish line for training. Training doesn’t end at the end of the course; it should also include follow-through and support. The [six disciplines] give us the framework to do that.”
The concepts are taking hold and paying dividends in terms of greater information; best-practice sharing across departments; and better overall program design, execution and evaluation.
Strengthening Follow-Through
In 2007, review of KP’s programs against the six disciplines criteria revealed Kaiser Permanente, like most learning organizations, was strong on the design and delivery of learning programs, but weak on support for follow-through, transfer and application. While Level 1 (reaction) data was collected routinely, there was limited data on behavioral changes or business impact.
To address these issues, NLL began to explore follow-through management systems. Just as learning management systems have made it possible to track courses, enrollment and attendance for thousands of employees and programs, follow-through management systems make it possible to track learning transfer and application for thousands of participants, programs, coaches and managers.
With the adoption of the six disciplines as a common learning language, NLL selected Fort Hill Co.’s Friday5s system as a follow-through management system that integrates disciplines three through six. NLL decided to pilot Friday5s with selected key leadership development courses. Once NLL reviewed the data, it determined that Friday5s created desirable follow-through metrics and the team decided to implement the tool on a broader scale. Now, courses at KP are developed with follow-through in mind.
During a core course, participants create goals for learning transfer by linking key concepts from the program to their priority job responsibilities and developmental needs. The system sends a copy of participants’ goals to their managers and coaches and encourages them to provide feedback. Over a designated period of about 10 weeks, participants are reminded biweekly by e-mail to update their progress by recording specific actions taken (Level 3) and outcomes achieved (Level 4). They are able to view each other’s reports to create a community of learners and seek feedback from coaches or managers on their progress.
Donna Lynne, president of Kaiser Permanente’s Colorado region, said, “The tool gives me the opportunity (as a manager) to review my nominee’s goals [for key executive leadership development courses throughout KP]. Once I review their goals, I either make suggestions or offer encouragement.”
Other KP executives expressed concerns that employees may resent working with a Web-based tool that requires biweekly updates on their learning goals; one worried that it might feel “Big Brother-esque.” Through extensive interviews, however, the L&D team discovered the opposite.
In fact, one KP participant and project manager said, “Receiving the e-mail reminding you to update your progress encourages you to work towards successful completion of your goals. I am not sure I would have achieved this level of success without the biweekly reminders.”
Another KP manager said, “I like seeing everyone else’s goals, and I especially enjoy finding other people working on applying the same skill sets. I knew I could call other participants for assistance.”
Broad Application
Following a successful pilot in 2007, NLL decided to adopt the same delivery and assessment system on a broad scale for core leadership courses. With participation up 500 percent compared to 2007, KP’s main challenge moved from understanding how to use the system to integrating follow-through effectively and obtaining measurable results.
The first challenge for KP’s rapid implementation was educating a diverse group of widely dispersed employees on how to use the tool. To assist people in the KP organization, the L&D organization created an internal Web site. Its features included advice on how to create a goal, current KP user metrics and participation rates. Webinar sessions are offered to trainers, designers, program sponsors and anyone who wants to learn more about the application.
Having mastered the normal operational activities of implementing and managing follow-through for 1,500 users, KP turned its attention to the challenge of maximizing the investment and demonstrating to program sponsors the positive impact it had on training.
Scoring and Tracking
The L&D organization developed scorecards highlighting high-level results from the tool. Specifically, these scorecards track key metrics such as participation by course and manager and coach response rates. The scorecards compare KP’s current participation stats with Fort Hill’s global data. Christopher Henry, KP senior manager, uses his scorecards to determine “where and how I can make improvements in subsequent sessions and future programs. I also use the cards to encourage current participants to update the system and complete their goals.”
Final scorecards are prepared for each course and provided to trainers and program sponsors. The scorecards list completed goals, as well as the incomplete goals or the goals not acted upon. This information is supplied to the KP learning leaders to support future improvement.
For example, the L&D group selects goals that were not completed or that were not clearly defined for further analysis. Interviews are conducted with the participants to determine what steps could have been taken to assist them in better defining or successfully completing their goals. Did they have adequate support? Did they receive proper coaching? What other assistance did they need?
Prior to the implementation of the follow-through management system, L&D had no insight into what happened following training and development. There was no way to easily identify either the success cases or the failures of implementation to drive continuous improvement.
The L&D organization also compiles quarterly reports for all trainers and course sponsors. The purpose of the quarterly report is to present the follow-through data in a consolidated report for easy analysis. These reports also are posted on the internal L&D Web site. KP employees may access the information to monitor their participation rates and compare them to similar programs in other regions or to Fort Hill’s global database norms. Management reports include data showing completion of goals, update rates and uncompleted goals for analysis to support future improvements.
Success Cases, Level 3 and 4 Measures, ROI
To document the impact of learning and development, Sachs’ organization reviews input in the database and selects individual success cases. The biweekly reports are reviewed for real-time evidence of behavioral change determined by actions taken and results, which are self-reported and noted in the manager’s feedback.
The team conducts follow-up interviews to determine to what extent the goal was achieved and how much value it contributed to the organization (Kirkpatrick Level 4). The L&D team has a series of steps to isolate the impact of the training program and determine the percentage of the goal attributable to the program.
The L&D team at KP is working with course participants to determine the ROI to the organization. For example, one participant’s goal was to “develop behavior guidelines for my work group so that we can hold each other accountable. Indicators of my progress will include improved collaboration and consistent behavior in the work group.”
Working with the participant, Sachs’ organization determined that this pharmacy manager and his management team spent an average of 120 minutes a day listening to and mediating employee behavior issues. Using the tools learned in his course, the manager applied the skills and worked with his team to develop behavior guidelines. The guidelines have resulted in increased collaboration among the team and have dramatically reduced employee interruptions to 25 minutes per day.
What does this mean to KP? Instead of mediating employee issues, this manager has more time to assist KP patients in the pharmacy and take care of issues relating to patient care. The results are tangible — saved time and labor costs — and equally important, intangible contributions further our mission of providing outstanding patient care.
Another example of a goal that is helping to improve the way KP does business is one participant’s pledge to “work through the steps for facilitating agreement in at least five meetings so that our meetings are more effective and meeting outcomes are reached 95 percent of the time.”
Prior to attending her course, this project manager discovered that poor communication during meetings resulted in one and a half wasted hours per week, or 68 hours a year. Once this individual set her goal and started following-through on the steps she learned in class, she achieved 95 percent of her meeting objectives. This represents a savings of 68 hours a year that this project manager is able to spend on productive and higher-value items.
KP has discovered the effective implementation of small goals can result in big savings to the organization. For example, a pharmacy manager attends a class, learns a new management technique and makes a small change to a process. This change results in a decrease in employee mediations. Subsequently, he saves approximately 95 minutes a day. Now he can use this time in the pharmacy assisting KP patients.
The L&D team would then determine the value of 95 minutes of his time, or the ROI for the organization. In this example, 95 minutes equals $718 in pay and benefits per day. The cost of the class is $2,000. The L&D team estimates that the isolated benefit from the course is 65 percent. Therefore, the cost savings to KP for this one change is $21,117 annually.
The Road Ahead
The L&D team is creating the KP-wide scorecard detailing ROI estimates based on individuals’ goals and accomplishments. The total of a large number of individual successes like those above will add up to significant savings and improved patient care. The key is to ensure that programs and goals are well-crafted, that there is a system to ensure follow-through and application, and that the results are recorded and rigorously analyzed. KP’s learning team knows that organizations improve one goal at a time, and it’s committed to driving individual changes that add up to large rewards and big improvements.
