Why Validity, Reliability and Job Analysis are Essential to Pre-Employment Assessments

key-pre-employment -psychometrics-.blog

Pre-employment assessments play a critical role in the success of every talent acquisition strategy. Talent acquisition (TA) professionals spend most of their days attracting and hiring top quality candidates into their organizations. Over the years, the science of psychometrics and the development of a wide variety of pre-employment assessments have played an important role in helping TA professionals evaluate candidates to ensure they hire the best talent.

When evaluating tests and pre-employment assessments for use in recruitment and selection programs, it’s important to understand the technical terms that vendors use to describe and market them. This is especially critical as test vendors sometimes use these important terms incorrectly. This can result in unintended consequences for both the organization administering the tests and the candidates taking them.

Here' how to interpret the three of the most commonly used terms in pre-employment assessments: validity, reliability and job analysis.

Validity: Do the Tests Work as Intended? 

Validity and reliability are two fundamental psychometric terms. Validity is really about the inferences we make from test scores. Technically, tests are not ‘valid’, but the hiring decisions we make based on test scores (hire-no hire) are either valid or not valid. Unfortunately, the term is often used incorrectly. It has, in fact, become standard practice for companies to say that their tests are valid.

Human Resources and TA folks expect to hear that tests are valid, so vendors tell them that they are.  That’s ok – they’re not industrial/organizational (I/O) psychologists. As an I/O psychologist, the most accurate definition for test validity I use is that the test does what it purports to do. That is, the test predicts who will be successful on a job or it indicates that someone has mastered some content area such as programming skills, etc. Tests are considered to be valid if the scores work as intended.

Reliability: Are the Test Scores Accurate? 

Test Reliability is an important term that is often used incorrectly as well. Reliability, in psychometrics (the science of mental measurement) refers to test score accuracy.

Every individual possesses an actual, stable level of a particular competency, trait or ability (e.g., math ability, information ordering ability, extraversion, etc.). In Classical Test Theory (CTT) this level of competence or ability is known as an individual’s true score. Tests are designed to give us an accurate estimate of an individual’s true score on a competency, trait or ability. When an individual takes a test – he or she receives a score. This test score is known as an individual’s observed score for a particular skill, trait or competence. This same individual may take the same test at a different time and receive a slightly different observed score. Or, the individual may take a different test that is assessing the same competency, trait or skill and receive a slightly different observed score on that assessment.

Observed scores always vary to some degree and are never perfectly stable. When an individual takes a test, it is theoretically impossible to obtain a perfectly accurate estimate of their true score – there will always be some degree of error in the measurement. Errors in measurement are inevitable and can be randomly caused by factors outside of the assessment such as fatigue, motivation, distractions during the test administration, etc. They can  also be more systematic and caused by properties of the test itself such as poorly written test questions. In CTT, reliability is concerned with the accuracy of observed test scores or the degree of error that goes into the candidate’s observed test score. In other words, how closely do test takers’ observed scores estimate their true scores.

Estimating Pre-Employment Test Reliability

There are many ways to estimate test reliability, one of the most common being to administer the same test after some time has elapsed and look at the correlation of scores. Typically, we look for correlations, which can range from 0 - 1, of .7 or higher as acceptable evidence of reliability. There are also internal consistency analyses where inter-item correlations are computed – but let’s save that discussion for Psychometrics 201!

Given this overview, you now know that a ‘valid and reliable’ test provides scores that are accurate and that can be confidently used for their intended purpose.

Job Analysis: The Foundation for a Defensible Testing Program

Underlying all of this for TA purposes, is the job analysis. A job analysis is a detailed study of a job that yields lists of the essential tasks required to perform that job. A job analysis can be conducted for a myriad of HR reasons, one of which is to ensure an organization is using appropriate tests when hiring. I/O psychologists and qualified test vendors or publishers work with an organization to conduct a job analysis for whatever position or job they need to hire. Once they have identified which job tasks are essential, they then determine the particular knowledge, skills, abilities and personality characteristics (KSAPs) that are needed in order to successfully perform those tasks. They then formally ‘link’ the KSAPs that are necessary for success to the job in question. Given this formal linkage, TA professionals can safely & defensibly use tests that assess those KSAPs.

Criterion-Related Validity and Benchmarking: Predicting Long-term Job Success

Going one step further, I/O psychologists can, and often do, conduct studies that show how well the test scores predict success at a given organization. This is a very powerful (defensible) type of study known as a predictive, criterion-related, validation study.

In this case, tests are administered to job applicants, who are then hired regardless of their test scores. Then after a period of time (usually 6 months to a year) test vendors and psychologists collect relevant work-related performance information (criteria) which might include supervisory performance ratings, number of widgets produced, sales volume, days tardy or absent, etc. for that same group of applicants or employees that were hired. We then compare the test scores to the criterion data and look at the relationship. Hopefully, we find that the scores were strong predictors of the variables we are interested in, and the test will be a useful component in the selection process – hence, predictive, criterion-related validation evidence.

Pre-Employment Tests: Must be Valid, Accurate & Job-Related

When selecting a talent evaluation vendor to assess candidates’ job skills and abilities, HR and TA professionals should ensure that the vendor provides tests that demonstrate evidence of validity (measure what they purport to,) reliability (accurately measure skills, competencies, traits, etc.,) and that are clearly job-related. Aspiring Minds pre-employment assessments have proven to be reliable measures that consistently predict job success.

Let's work together to help you identify the appropriate pre-employments assessments you need to hire the best talent. Get in touch today. And please reach out to me if you have questions about how valid and reliable tests impact your hiring process. Connect with me on LinkedIN.  I'm here to help.


About the author

Fred Rafilson, the Chief I/O Psychologist at Aspiring Minds, works with clients to ensure their testing programs are valid, reliable and fair. He has developed and published over 30 pre-employment exams and serves as an expert in employment litigation cases.  Contact Fred at Fred.Rafilson@AspiringMinds.com

Get In Touch