Despite persistent emphasis on more sophisticated learning analytics, post-learning satisfaction surveys — often derisively called “smile sheets” — continue to be the most common method of acquiring feedback on learning initiatives.
A majority (78 percent) of learning executives surveyed indicated they continue to use satisfaction surveys for post-learning assessment, according to a report analyzing survey data from the Chief Learning Officer Business Intelligence Board (BIB) set for release next week.
In January, the editors surveyed the BIB, a group of nearly 1,500 professionals in the learning and development industry, to assess and benchmark learning analytics practices. The full analysis of the results will be released in “Focus on Learning Analytics,” a Chief Learning Officer report.
In addition to smile sheets, other common post-learning assessments include formal testing, reported by 41 percent of the BIB, and anecdotal observation, reported by 36 percent. While these remain the most used methods, the BIB reports their use actually decreased from 2009 to 2010. In 2009, 86 percent of survey respondents indicated they used satisfaction surveys; 49 percent used formal testing; and 45 percent used anecdotal observation.
Many practitioners and consultants in the learning and development industry, including this magazine, have lobbied for more sophisticated measurement methods that go beyond use and satisfaction to the effectiveness of learning, its impact on employee productivity or its ultimate impact on organizational goals. According to BIB survey data, that movement remains a work in progress.
A majority of respondents (61 percent) report learning output — such as number of courses, number of students and hours of training — to executives in order to demonstrate learning’s impact on the enterprise. Nearly 49 percent reported student or stakeholder satisfaction, but only 9 percent reported impacts on employee productivity.
Output and satisfaction are relatively easy to measure, which is why many enterprises frequently use them. But their continued use has not significantly improved overall satisfaction with learning measurement efforts. When BIB members were asked how satisfied they are with their organization’s L&D measurement, a minority said they were satisfied or very satisfied (37 percent). Nearly 40 percent indicated that they were dissatisfied or very dissatisfied with measurement.
Delving further into the data provides a possible explanation for the continued dissatisfaction. More sophisticated measurement and data that are meaningful to executives outside the learning department remain a work in progress.
According to survey results, 51 percent of BIB members correlate job performance back to learning, with a further 23 percent indicating they have plans to put in a performance measurement plan within the next two years.
About 36 percent of BIB members reported that they correlate learning to their employees’ productivity, with a further 23 percent indicating they plan to do so in the next 12 to 24 months.
Approximately 39 percent of survey respondents correlate learning to overall business performance, with an additional 30 percent planning to correlate performance and learning in the next two years.
The reasons for this continued lack of satisfaction with measurement and the lagging sophistication in methods are parallel. BIB members most commonly reported a lack of resources to engage in measurement and a lack of management support or interest.
This leaves learning leaders in a quandary. They remain unsatisfied with their current measurement efforts but report they are unable to get the resources or support to embark on more sophisticated measures that would be meaningful to the business.
But the longer learning leaders rely on smile sheets and other output and satisfaction measures, the farther they’ll be from acquiring more meaningful business metrics. Smile sheets are dead. Long live smile sheets.
