Choosing the Right Tool: A Process for Evaluating Learning Technology

Some details in this post have been adapted to protect confidentiality.

The first time I was asked to recommend a software purchase, I didn’t have a process for evaluating the available options. Stakeholders wanted LinkedIn Learning-style videos, but without building a set or hiring on-screen talent, so they asked me to explore AI platforms that could produce talking-head videos instead. Because of the attention AI was getting, this felt like more than a typical tool choice. It was a real chance to explore what AI could do and whether it was ready to play a larger role in development.

To make the decision, I used a four-step process: define the need, gather options, evaluate the shortlist, and make a recommendation.

Defining the Need

I started by clarifying the outcome the team needed: who would use the tool and what they had to do with it. The priorities were high-quality instructional videos, produced quickly and at low cost, on a platform that was easy to learn and SCORM compliant. Protecting proprietary information and IP was non-negotiable from day one.

Gathering Options

The second step was to  identify vendors that could meet the business need. I began with a wide range of options and quickly eliminated several based on the non-negotiables, which left a shortlist of two.

Evaluating the Shortlist

Next, I needed to define evaluation criteria for making a final recommendation. For this project, the criteria included cost, quality, speed, ease of updates, and platform innovation. I used trials and sandboxes to understand each tool’s capabilities, and I spoke with sales representatives about how each platform aligned with our business needs. Throughout the process, I looked for strengths, weaknesses, and differentiators.

Going in, platform innovation looked like the clearest differentiator. Platform A offered L&D-focused features that Platform B didn’t. In practice, much of that functionality wasn’t useful to the organization, so the criterion I expected to make a difference ended up carrying less weight.

The deciding factor came from testing SCORM packages from both platforms in the LMS. At the time, Platform A’s interactive courses used a dynamic HTML launcher that embeds the video through an iframe pointing to an external URL. The video assets aren’t stored inside the zip file. That raised two concerns. The first was learner privacy. The second was what would happen to the content if the organization were to end its relationship with Platform A.

Making the Recommendation

The platforms were comparable in most other areas, and Platform B was significantly more cost-effective, so I recommended Platform B. My recommendation included the software's purpose, the competitive landscape, my rationale, and next steps.

Ultimately, with Platform B, we shipped three video-based courses at a fraction of the cost. We halved production time, and stakeholders responded positively.

Takeaway

This decision played out at a small scale, but the same four steps apply to any tool decision. For larger purchases or tools used more widely across an organization, each step expands to include more stakeholders, more use cases, more vendors, possibly an RFP, and a more rigorous recommendation. But the structure stays the same.

The process is essential because the feature that looks most impressive in the marketing materials isn’t relevant to the organization’s business need. Testing the tool in your own environment is how the final decision ultimately happens.