Using Analytics to Track the Effectiveness of Your Instructing

You could have the best eLearning suite, but if you don’t have robust analytics, it won’t matter. Data is imperative for effective content delivery as it offers clean and comprehensive insights into how your learners are, well, learning.

How Do We Qualify Learning?

Learning is a nebulous concept, despite people trying to systemize it, we still don’t know the best techniques or ways of measuring it. Measurements like IQ scores have shown again and again to have little to do with the innate intelligence of people (IQ scores are dramatically affected by familiarity with IQ tests).

All of this is a long-winded way of saying we don’t know how to exactly measure learning, but we can utilize data for visualizing trends. For example, if we plotted the scores of a learner’s test scores, it would be impossible for us to determine how much they learnt, but we could infer that they are learning from an increasing trend in scores.

All of this begs the question: what are the best learning metrics to track?

The Best Learning Metrics to Track

Deciding the best metrics to track will depend on your goals, but for the goal of learning, these are the best themes to follow: time, engagement, and progress. It is best to think about these themes as pieces to a formula, like so:

As this formula represents, time, engagement, and progress all have a positive correlation with the learning potential. However, there are some caveats, but we’ll discuss every piece in detail below.

Time

This piece has the biggest and most noticeable caveat. Time is logarithmic in learning as, past a certain threshold, the amount of time spent learning will not be as effective as it once was. Yet, it is still undoubtably one of the most important metrics of which to measure learning progress.

Measuring time can be done in multiple different ways, but the most important should always be time spent in course. Time spent online or outside of a course can easily not be representative of actual time spent learning, as one can easily idle in either of these situations.

Engagement

Engagement is perhaps the trickiest metric to properly measure as it can be broken down into multiple actions, from how often a learner posts on discussions, attendance or if they’ve completed surveys. Thankfully CoreAchieve tracks metrics like these, so it saves you from manually counting.

Engagement should also be rated on the quality of work too, especially with written responses like discussions. A learner may look like they’re engaged on a graph, but their responses could all be nonsensical.

Engagement also will vary depending on the tools you have available. CoreAchieve, for example, collects data on every tool offered including flashcards. If a learner is consistently using flashcards and improving with them, then it is safe to assume they are engaged with the content.

Progress

Progress is the most traditional way of gaging how much someone has learnt, referring to scores or levels of completion. This metric has the most sway over the amount of learning potential as the other two can be misleading if someone already knows the content or is doing the bare minimum, whereas it’s very difficult to fake passing a test.

Yet, there are still something to consider. Namely, what type of questions are asked, are answers shown after failing, and how many retakes can learners take. All these factors, especially whenever combined, allow learners to game the system and make it look as if they’re progressing whenever they are not. Keep an eye on how many times a learner takes an assessment to determine how they achieved their progress.

Learning Potential

Learning potential is a mixture of all components, as they their efficiency depends on one another. For example, progress (depending on how assessments are set up) can be gamed to run through assessments as quickly as possible, but if you factor time into the equation, it’ll be easy to tell how legitimate a learner’s progress is. Likewise, high engagement but low time spent is another sign of just doing the bare minimum for completion.

Learning potential also has the caveat that it does not reflect one-to-one to how much the user has learnt. However, it does allow you to estimate the effectiveness of your learning.

The Best Way to Visualize This Data

Now that we’ve discussed which data to track, we need to find the best way to visualize this data to make it useful. For time, it would be important to have all the learners’ times next to each other so you can immediately see any possible discrepancies. This approach will also help you determine if a low time is abnormal or if every learner could be on par.

That idea follows through with all data types. Essentially you want to be able to look at all learners’ progress to discover trends, but you’ll also need to view your learners at an individual level. By looking at a learner’s data individually, you can determine metrics like engagement more effectively than scouring everyone’s work.

Not All Graphs Are Born the Same

So, you’ve collected all this data and want to graph it to make it easier to read, but some graph types can be misleading. Before making a graph, determine what the goal of the graph is. Do you want to show proportions or visualize trends? Graphs, like everything tell implicit stories.

Pie Graphs – It’s best to avoid these unless you are measuring a based engagement overall like: How many users opened this optional module.

Scatterplots – This is the best way to visualize trends of individuals and groups. Make the x-axis some type of linear time (quiz attempts or courses) and the y-axis their scores.

Line Graphs – Not too particularly needed here, but you could use it to visualize interactions with many optional courses to find what your most engaging content is.

Bar Graphs – Useful for measuring more than two axes, namely with the stacked bar graph. For example, you could have the x-axis be learners, y-axis as completion rate, and have the stacks on the bar be different courses they’ve been assigned.

With all of these, you must be careful. It is very easy to make correlations that don’t mean anything and complex graphs that aren’t worth deciphering.

In Summary

The three major metrics for learning potential are:

• Time – Logarithmic, after a certain point time sees diminishing returns whenever it comes to learning. The time that matters most is time spent inside the actual courses, not just log-in time.

• Engagement – Admittedly the trickiest one to properly measure, engagement refers to how the learner interacts with the content. Are they finishing courses, participating meaningfully in discussions, or using extra content like flash cards?

• Progress – The most telling metric of all, progress is the oldest and most common way of measuring learning potential. This refers primarily to grades over time, but you should be careful with automatic systems as progress can be gamed to maximize output while putting very little effort in.

Regardless of how you go about measuring your learner’s progression, you’ll want a strong set of tools to do the heavy lifting for you. CoreAchieve provides this with Microsoft’s Power BI, generating and graphing automatic reports on all the data you would need. Get started with CoreAchieve today for free.