Three Critical Considerations for Evaluating Talent Assessments Today

Over the past several decades, many organizations have turned to psychological testing and assessment methods to improve the talent selection process. Not only do these tried and true methods increase objectivity and prediction, but they have been well-researched and identify reliable measures of ability, personality, values, interests, knowledge, and skill.

Today, with the rise of machine learning/artificial intelligence (AI) capabilities and increased attention on the candidate experience, new, technologically-driven selection methods are emerging. While these methods offer exciting new opportunities, it is important to ensure that any assessment solution adheres to industry-accepted standards. One way of doing that is to be thoughtful about how we build these new solutions, using the expertise of industrial-organizational (I-O) psychologists to inform their development.

Here are three key considerations from the I-O world for evaluating new (and existing!) methods of talent assessment:

1. Understand what you’re measuring with talent assessments

At the risk of oversimplifying AI’s benefits, it primarily helps us with prediction. And prediction is hard, especially when it comes to human behavior. Whether it’s your manager, direct report, friend, spouse, child – it’s frustrating when their behavior doesn’t align with your expectations. With the proliferation of data and AI capabilities, we are now able to build models that increase our ability to predict behavior in areas like employee retention, employer net promoter score (eNPS), and quality of hire in the selection process.

However, just because we can better predict behavior, do we know why? Whether you’re evaluating thousands of data points from a recorded interview, parsing resumes, or scraping someone’s digital footprint, do you know what exactly you’re measuring? Whether it’s customer service behaviors, empathy, key experiences, or teamwork, be sure you’re aware of what you’re measuring and why it may predict performance. It’s important to be able to explain why a prediction is relevant to a particular outcome. And it’s important to recognize that some new methodologies do not yet produce assessment scores with the same reliability and validity as traditional assessment methods. Understanding what you want to measure may determine how you go about measuring it.

2. Ensure your talent assessment results are consistent and precise

One trend related to enhancing the candidate experience is gamified assessments or game-based assessments. These methods are appealing to both hiring managers and candidates because it avoids having assessments “feel” like a test. Gamified assessments can screen candidates in an engaging way and can also be used as a tool to attract candidates. Incorporating gaming elements in an assessment process can provide opportunities to measure characteristics, behaviors, or skills we wouldn’t otherwise measure with traditional assessments.

However, when evaluating these “fun” assessment experiences, it is important to understand what they are ultimately measuring. If you take a game and it indicates you’re outgoing and resilient, then it should reflect your personality and be consistent with other methods of assessment that measure those same characteristics. If your gaming results don’t make sense or are inconsistent with other assessments, then it may be a fun game, but it may not be very reliable at what it aims to measure. Furthermore, you should receive consistent results if it is measuring stable characteristics (i.e., if you take the game multiple times, your score on stable characteristics shouldn’t change much). For job candidates that only get a limited amount of time to perform well in the selection process, all methods should measure job related characteristics (e.g., personality, ability, skills) in a precise and consistent manner.

3. Be sure your talent assessment approach works for your organization

Each organization has its unique employment brand, hiring needs, and talent pools. When incorporating any method of assessment, it’s important to understand why it may (or may not) work for your specific organization. If we can’t explain what is being measured and why it relates to job performance (or another important outcome) then why would we expect it to continue to predict outcomes across jobs, industries, and organizations? These explanations might also need to be given to a regulatory agency or a candidate, so it’s particularly important to understand the process. For emerging methods that may not yet have received the same amount of broader empirical research as traditional psychological assessment methods, it may be necessary to conduct a study within your organization to help provide explanations and additional support.

There are tremendous and exciting opportunities with both emerging and existing assessment approaches. However, to promote fair and effective selection decisions, emerging methods should be evaluated with the same professional and legal standards as traditional psychological assessments. Be sure you are getting the best of both worlds when selecting your selection approaches.