Measuring whether training worked: four levels of evidence

Why completion is not a result

A learning system produces numbers easily: who enrolled, who finished, how long it took, what score they got. Those numbers describe the training. They do not describe the work. A person can complete a course and change nothing about how they do their job, and a person can change how they work without finishing the course. The first step in measuring training is to be clear about which question each number answers.

Completion figures also need care when they are compared with outside numbers. Jordan (2015) gathered enrolment and completion data for 221 massive open online courses. Completion, defined as the percentage of enrolled students who completed, varied from 0.7 percent to 52.1 percent, with a median of 12.6 percent. Longer courses had lower completion and courses using auto-grading only had higher completion.^[1] That is evidence about open courses taken voluntarily, where nothing is at stake for the learner. It is context, not a target for training a company assigns to its own staff.

The four levels

The Kirkpatrick model, widely used in the training field, sets four levels of evaluation: reaction, learning, behaviour and results.^[2][3] They are a useful checklist because each level asks a different question.

Reaction asks how people felt about the training. A short survey after a module can capture relevance, clarity, pace and confidence to apply the material. It is the cheapest early signal and it can show a module that confuses people.^[2][3] It does not show that anyone learned anything. In the testing studies on student memory, learners felt more confident after restudying than after being tested, yet remembered less later, so how a method feels is not a reliable guide to how well it works.^[4]

Learning asks whether people can now do or know something they could not before. A test measures it.^[2][3] Beyond a pass or fail, a learning system can hold attempts to pass, time on module, item-level difficulty and score distributions, and those can show which questions or modules are weak.

Behaviour asks whether the work has changed. This is the level an owner usually means by "did it work", and it cannot be read from a learning system alone. It needs a measure taken from the work itself.

Results asks whether the business outcome moved. It is the hardest to attribute to training, because many other things change at the same time.

Making the test worth reading

A score is only as good as the test behind it. The quality-assurance sources on assessment name two properties: validity, meaning the test reflects the learning outcomes of the module, and reliability, meaning results are stable and not noise.^[5] The pass mark is a policy decision. Standard-setting methods exist to choose a cut score that is justified and then checked against actual results, and the choice is better recorded with its rationale than left as a default.^[6]

Where assessors mark work by hand, moderation is a further check: different assessors marking the same work should reach consistent judgements, and the process can reveal ambiguous questions.^[7] These are practices from education and training quality assurance. They are offered here as a vocabulary for asking questions of any provider, not as a standard that applies to every course.

For reference, one live product states its rules plainly: The Works LMS, which we make, uses a pass mark of 75 percent, allows two attempts per assessment, and draws final-exam questions from a larger bank. Those rules tell a reader what a pass means. They do not say whether work changed, which is a level three and four question.

Learning data: what a system can hold

The standards behind learning data describe activity as records of who did what. The Experience API, known as xAPI, records statements in the form actor, verb, object, such as a named learner completing a module, and stores them in a learning record store.^[8] Caliper Analytics, from 1EdTech, describes events using controlled vocabularies for activity types such as assessment, media and reading.^[9] Learning analytics tools commonly report engagement measures such as login frequency and time on task, and completion measures such as module completion, quiz submission, time to completion and pass rates.

Academic work on predictive models reports that behavioural features such as login frequency, quiz submission and task completion can help flag learners likely to disengage.^[13] Accuracy and retention figures from such studies come from particular institutions and should be validated against a provider's own learners before they are quoted as targets. The practical point is narrow: data held about activity can help spot where learners struggle, and it still stops at level two unless it is joined to a measure from the work.

Choosing a measure from the work

A measure for level three or four has to come from the workplace and be specific to the task that was trained. The examples below are illustrations of the idea, offered as a design suggestion, not as findings or benchmarks. For a sales team they might be the share of enquiries answered within a set time, follow-ups logged after viewings, or listings published with a permit number recorded. For an office they might be errors found in a monthly file check or the time from instruction to a live listing. A good measure is one the person doing the work agrees is part of the job, that can be counted without much effort, and that does not depend on one person's opinion.

Timing matters as well. The research on retention found that benefits of testing and spacing show up when retention is measured after a delay, so a measure taken the day after training says little about whether it lasted.^[4] Taking the figure before the training, then again after a month and after a few months, gives a trend. The right interval depends on the task.

Comparing groups fairly

A natural step is to compare people who took the training with people who did not. The evidence base on online learning shows why this can mislead. In the US Department of Education review of online learning studies, the authors cautioned that conditions often differed in curriculum, pedagogy and learning time, so an advantage may not be rooted in the medium.^[10] Means and colleagues found that blended studies tended to include extra time, resources and learner interaction, which confounds the comparison.^[11] Cook and colleagues reported large heterogeneity across 201 studies of Internet-based learning, meaning that an average hides wide variation.^[12]

The lesson for an employer is to ask why the groups might differ before crediting the training. People chosen for a course may already be more engaged, and teams that train together share a manager and a market. A before-and-after comparison within the same people is simpler to read than a comparison between groups, though it cannot rule out other changes over the same period.

What the evidence does not show

None of the sources read reports a controlled trial of employer-assigned training measured at levels three and four, and the learning studies cited here mostly concern students. The numbers above should be read as context for asking good questions of a provider, including asking for its own completion and outcome data, and not as targets. This page is general information.

Questions