By: Josh Bersin

                                                                   March 1, 2002    e-learning Magazine

 

E-learning can be a powerful tool-it is scalable, less expensive than traditional training, and in some cases, it can really transform a business. But after spending a lot of money on infrastructure and content, how do you know if your e-learning content or program is really effective?

In this article I review a proven methodology for measuring effectiveness, which is the first part of really measuring return on investment (ROI). Over the years of deploying large e-learning programs at DigitalThink (www.digitalthink.com) and other companies, I found that this process works flawlessly, and is easy to understand. You can use this methodology to measure:

Remember what effectiveness means

Remember that the ultimate purpose of e-learning is not to reduce the cost of training, but to drive business results. If you can't identify the business goal of an e-learning program, you have to ask why you are doing it in the first place.

Unlike traditional instructor-led training, with e-learning you don't have the ability to look students in the eye, talk to them at the break, and get direct feedback on whether they're really learning anything. Instead, you have a lot of data. Wisely using this data is the key to determining effectiveness and getting a return on your investment.

I group e-learning effectiveness into five steps-(the "Bersin Five Step Program?"). Remember that for your analysis to work, you will have to measure each of these steps using some form of data that you get from your learning management system (LMS), from your hosted content provider, or from your content delivery system.

There can be a whole set of issues in getting this information, but if you are using industry-standard AICC content, in most cases you will be able to find this information.

Step1: Enrollments. "Is the audience showing up?"

The first and most obvious thing to measure is enrollments. Is the audience actually enrolling in the course or courses? In most systems, courses are stored in your dynamic course catalog, and students must proactively enroll in order to start a course. In some cases they are "pre-enrolled" or "automatically enrolled."

One valuable exercise is to monitor enrollments. If a course is launched on February 1, and the audience size is 1,000, what percentage of the people have enrolled by February 10? Try to monitor enrollments weekly.

If learners are not enrolling, then you probably have a marketing problem. People either can't find the course, they don't know how to enroll, or they do not understand why it is important. You may have to engage in a more active marketing program.

If the course is a "directed learning" program-one mandated by management-then enrollments should pick up fast. If it is "elective," then perhaps the course is named poorly, not well positioned in the catalog, or people do not even know it exists. You have to take on the responsibility of marketing your e-learning programs-it is very easy for people to ignore the program completely if they don't understand why they need it. See figure 1.

Step2: Activity. "Are they eating the dog food?"

The next important issue to address is "are people moving through the course?" Have they started? What "percent completion" have they achieved?

With some content, you may not be able to identify "percent complete" because the content was not built to support this level of detail. You should demand that "percent complete" be built into any vendor or custom course-and tracked at every chapter, lesson, interactivity, or assessment.

You should monitor activity correlated to enrollment date. For example, if you take a group of people who enrolled the first week of February, how far have they progressed by the end of February? New enrollments will not progress as far as prior enrollments. If you find that people are enrolling but not getting very far, then you have a content problem.

Maybe the content is too hard to read, it runs too slowly, or it may just be boring or difficult. Typically, if the content is appropriate for the audience, and you have the right content for the right business strategy, people will progress at a reasonable rate.

What is a reasonable rate? Well, it depends. Typically if a course is a few hours long, you will find that people progress at an hour a week or so. You may find that people go quickly and then stop at a particular point. This information is very valuable to help you assess the usability, relevance, and performance of the content.

Step3:Completion. "Did they finish?"

Completion is just a case of "activity." But it is a very special case. People who truly complete, meaning that they actually take the entire course, deserve special recognition. These are the people who will give you the best feedback on content quality and effectiveness toward the business goal. They wanted to finish.

By the way, be careful here. Many courses will flag a student "complete" if he/she touches the final lesson-even if he/she skips everything in between. If this happens, you have a course that was not well designed. Again, you should demand from your content providers that they provide you a true completion-meaning that the course measures whether a student actually touched every chapter or every page.

Another interesting point: completion percentages are not something you can "average." A bunch of students that achieved 30 percent completion is not the same as one-third of the students who achieved 100 percent completion. The former means that you have a poorly performing program. The latter means that you may have a great program, but it's not targeted very well.

Targeting your program is important-it's like marketing. If you target the right ontent toward the right audience, you will achieve the desired business result.

 

 

 

Step4:Scores. "How well did he/she scores? 

Many people think this is the one and only way to measure effectiveness. I tend to disagree. Sure, if people score highly, they have learned something. But in e-learning you don't always know why they scored high. Did they really learn the material or did they copy from someone else? Or did they already know the material? Or did they just try the test 15 times until they got it right?

Here there is another technology issue. Does the course count the number of times a student attempts a question? If not, the score data may be useless-because people will keep guessing until they get it right.

Also, are scores only taken at the end or are there assessments along the way that you can use to measure learning? It is important to have multiple assessments, so you can measure progress toward the final learning goal. And the best content actually will categorize assessments by learning objective, so you can measure what exactly someone has scored well on and where they have fallen short.

Score and completion percentage tell you a lot. You will see that people fall into different segments based on completion percentage and scores. See figure 2.

Step5  :Feedback/Surveys. "Did they like it?"

 

Feedback is a vital part of e-learning. Remember: unlike traditional training, you have little or no face-to-face contact with learners. You should regularly request and enable learner feedback-in numeric and written form.

Feedback will tell you important things like: Did the content play? Did the assessments work? Did the video, audio, and other media work? Were the material and interactivities engaging? Was the material useful? Were the graphics informative?

You will find that the more "real personality" you put into e-learning, the more effective it will be. For example, if you have a "graded assignment" that goes to a mentor or tutor, you will find that course completion goes up by orders of magnitude. If you make live, synchronous sessions mandatory (with attendance monitored) you will find that people get their pre-work done. There are many ways to incorporate real people into e-learning, and these have a huge effect on effectiveness.

Over the course of your e-learning experience, you will spend a lot of your time and money on content. If you do not measure these five things, you will never know if you are getting your money's worth. I also guarantee that at some point an executive will come to you and specifically ask, "What are we getting for all this e-learning, anyway?"

Measurement is important. As Peter Drucker once said, "if you can't measure it, you can't manage it."

If you think about measuring effectiveness early and often, you will find that your entire e-learning program is cost effective, powerful, and aligned with the business. This is what e-learning is all about.