Learning strategies are an integral part of any enterprise knowledge management (KM) strategy and must be tied with content management, search and portal and collaboration frameworks. In order to achieve this level of organizational value, both technology and business owners must ensure that learning investments are aligned with business directions and workforce management metrics as well as compatible enterprise-wide technology infrastructure. This level of business agility will facilitate externalization of learning services to a broader constituency of customers, partners and suppliers.
Performance of the learning management system (LMS) market has not matched original expectations. Recent research from the LMS METASpectrum (February 2003) indicates that current economic conditions and a high incidence of LMS implementation failure have led to a decline in the annual growth rate from 40 percent in 2001 to 10 percent in 2002. We also believe that a lengthy implementation cycle (the average LMS implementation lasts three years) also has contributed to diminished user satisfaction. The research indicates that the following factors are common when organizations abandon an LMS solution:
- Lack of defined business objectives and ROI measures (e.g., purchasing an LMS just because the organization has a goal to become more training-focused).
- Over-customization as many organizations choose an LMS without considering usability, automated processes and realistic budgets, resulting in increased development and implementation costs in an attempt to make a poorly chosen offering more palatable to users.
- Lack of strategic focus and poor planning (e.g., departmental decision without thought to enterprise reuse, lack of integration and alignment with HR systems, focusing on intangible measurement instead of more pragmatic ROI tied to specific business initiatives).
The learning management system (LMS) market was $473 million (U.S.) in 2002, consisting of LMS/LCMS technology ($142 million), professional services ($285 million) and performance management ($48 million). (See Figure 1.) These numbers represent the shift to a vertical initiative-focused market with vendors conating to create suite solutions spanning learning content management systems (LCMS), course development and skill management (e.g., for learning object repositories, testing, local language translation, skills mapping and succession planning).
LMS Vendor Assessment
Making the right decision on an LMS provider requires a formal decision process. We recommend organizations begin with identifying business requirements and then perform a a self-evaluation to identify where they are in terms of learning maturity and what is needed to achieve learning excellence from both a process and cross-functional perspective. Within this self-evaluation, organizations should also assess current technology capabilities (gap analysis), mapping them to their organizational requirements to facilitate closing those gaps over time. When making LMS choices, decision-makers should realize that an LMS is not a starting point for a blended training strategy. Before making an LMS decision, organizations should gain experience using e-learning delivery technologies, such as live virtual classrooms (e.g., webinar) for instructor-led training, asynchronous virtual classrooms and electronic performance support systems (EPSS) and other self-service knowledge bases for 24×7 training and resource support. Research indicates that many organizations follow a specific maturity process as they become proficient in blended-learning strategies and tie this to an integrated knolwedge management infrastructure. (See Figure 2.)
Figure 2: E-Learning Maturity Model
Maturity Level | Strategic Inflection Point | Technology | Critical Behaviors |
1. Organic | Learning is at the individual level for skill attainment and professional development. | ILT, CBT, CD-ROM | Ad-hoc, reactionary response to individual staff demands. |
2. Initiative-Driven | Learning becomes more systemic, with repeatable methods and practices around specific business strategies (e.g., CRM, HIPAA) funded through strategic initiative. | Synchronous and asynchronous delivery platforms, virtual classrooms | Learning is targeted at process outcomes. Objective is business-unit excellence. |
3. Enterprise-Based | Disparate learning strategies are conated and centralized within the organization, enabling consistent life-cycle learning management. | LMS, LCMS | The organization can establish consistent learning goals and objectives between business units. Reuse of IT investments via shared infrastructure. |
4. Competency-Based | Learning becomes part of broader, enterprise-wide workforce management framework. | Integration is complete (bi-directional) between learning and HRMS platforms learning generates workforce analytics for scorecards and benchmarking. | Learning becomes massively customized to meet end-user roles, performance needs and individual/teamwork models. |
5. Knowledge-Management-Based | Learning strategies intersect internal knowledge management efforts they are externalized to partners and customers. | Integration with portals, enterprise content management and collaboration tools interfaces to payment systems for monetizing learning applications. | Human capital management and knowledge management strategies are unified, and learning becomes a key element to enhance workplace performance and innovation. |
Once a self-assesement is done, a vendor list must be identified and evaluated via a structured methodology for accelerating the understanding of vendors’ capabilities as they relate to the organization’s business requirements. META Group’s METAspectrum is an example of such an assessment tool that provides a decision framework that also enables organizations to modify the inportance of the evaluation criteria they feel are important to their investment decisions. The METAspectrum methodology uses evaluation criteria and sub-criteria that are tailored and weighted to particular market characteristics and plotted against performance and presence factors. Performance characteristics include product and service areas that represent the ability of the vendor to provide value to customers, such as pricing and financials. Presence charactersistics focus on on how the vendor delivers value and can be measured by subjective criteria, such as business drivers and partners/channel delivery strategies. (See Figure 3.)
META Group’s February 2003 METASpectrum LMS market evaluation rated 13 LMS vendors that met the criteria for inclusion of more than 20 live implementations of more than 20,000 learners, relative financial stability and positive references from within our client base. It should be noted that IBM/Lotus, Siebel, SAP and PeopleSoft did not meet the inclusion criteria, but META Group believes these vendors are poised to move in and compete with best-of-breed LMS solutions (e.g., Docent, Saba, Intellinex, Plateau and Click2learn), causing further vendor conation over time.
Particular evaluation categories that organizations should look for in examining LMS vendors are tied to market presence and performance:
- Financials: Vendor viability is a rational concern, given that the learning management vendor landscape has been in a state of contraction.
Focus: Organizations should look to the selection of an LMS vendor as leading to a long-term partnership and should establish a periodic review schedule to re-evaluate the viability of the vendor over the long term.
Best Practice: Organizations should involve their CFOs and require that the LMS vendor make full disclosure of important financial metrics as a prerequisite to selection. For private companies, this information should be gathered via NDA agreements.
- Pricing: Should include evaluation of pricing schedules, discounts, maintenance, the entire life cycle of purchases, as well as the value (e.g., ROI) of strategic business (e.g., innovation, CRM, ERP, risk management) that the vendor has delivered to clients in the recent past.
Focus: Careful attention must be paid to maintenance agreements and terms and conditions. Moreover, many Global 2000 organizations migrate from a hosted to an installed option. Consequently, vendors should offer pricing options for both hosted and on-premise solutions and have experience with both scenarios.
Best Practice: Contract and final pricing negotiations should be conducted with at least the two leading candidates to obtain maximum leverage.
- Business Vision: Vendors must keep ahead of the market through building and maintaining core LMS competencies, intellectual property and a corporate culture of excellence, and by focusing on targeted growth areas.
Focus: Organizations must check the alignment of the vendor’s business drivers with their own corporate culture. Business drivers must also address the vendor’s ability to handle an organization’s learning drivers.
Best Practice: Reference calls (three are recommended for each vendor) and/or end-user site visits (two are recommended for each vendor).
- Channels/Partners: Large systems integrator awareness of an LMS vendor’s technology and processes is crucial to enterprise implementations, as most vendors’ professional services often encounter problems with implementing technology solutions that involve complex processes requiring major organizational changes.
Focus Area: When making e-learning enterprise decisions, leverage existing organizational relationships with systems integrators to discuss strategy and implementation assistance. However, most systems integrators will also have a formal relationship with one or more vendors. Clients should be fully aware of their ability to dispense impartial advice as the systems integrator may have investments within specific vendors.
Best Practice: Learning management systems with strong channel programs with defined methodologies must ensure the quality, consistency and value delivered by individual partners. This is particularly an issue around system integrators’ disengagement (e.g., how to manage processes after a consultant leaves).
- Technology: Evaluations must be tied to the specific needs of the organization with the focus on integration with existing applications.
Focus: Critical factors include open architecture, standards certification, usability, ease of integration and automated handling of traditional training/administrative functions. Depending on requirements, learning content management, authoring capacities and collaborative teamware differ by vendor.
Best Practice: The vendor’s ability to demonstrate an open architecture, third-party integration and e-learning standards compliance (particularly interoperability with the organization’s existing e-learning content) is critical.
The Future of the LMS Market
E-learning must be viewed as a key juncture between workforce management and knowledge management, but should not be exclusively tied to either. Organizations failing to establish leadership integrity and workforce commitment will face steady erosion of products and services (quality, R&D, defect rates, market share) resulting from employee malaise. Enterprises should establish foundational elements for improving employee commitment now by funding organizational improvement programs, such as blended training initiatives.
During 2003 and 2004, look to companies that are not e-learning-specific (e.g., IBM, SAP, PeopleSoft and Documentum) to participate more actively in the market, as e-learning software becomes an integral part of global enterprise deployments. While this will cause further vendor conation, it will not put best-of-breed LMS vendors out of business in the near term, although many vendors will eventually look to become departmental systems to service small to -sized businesses by 2004.
Jennifer Vollmer is a lead analyst for learning management for the META Group. For more information, visit www.metagroup.com.
