The Real Multi-Layer Defence against Cheating

The rise of ChatGPT has caused an upsurge in concern about candidate cheating in online talent assessment, as well as across educational assessment more broadly. There is no doubt that ChatGPT and other generative AI tools and AI assistants are spookily impressive in what they can do. Equally, neither is there any doubt that it is a long-established best practice in talent assessment to design assessments that can counteract and prevent cheating.

At Cappfinity for a number of years we have used the principles of the Cappfinity 3D Model – Design, Deter, and Detect – to describe how we have implemented the range of interlocking, multi-layer approaches to support the prevention, deterrence, and detection of cheating, starting with the fundamentals of assessment design.

As we think about candidate cheating, it becomes clear that there are typically three ways in which a candidate might cheat on a talent assessment:

• Having illegitimate access to the ‘right answers’ for the assessment

• Having another person complete the assessment on your behalf

• Using an AI assistant or other tool to provide the answers on your behalf.

It’s worth noting that this is all quite different from how ChatGPT is being reported in the media as helping people with job applications. Media reports are focused on the simple premise that job applicants are using ChatGPT to help them write better application forms or cover letters.

Hold on a minute, isn’t that what careers advisers, parents, and helpful aunts have been doing for years and years? If anything, ChatGPT has simply levelled the playing field, giving more people access to more powerful technology to help them.

Applied to talent assessment though, there is a different question to answer. Talent assessment is focused on getting to know the psychology of who you are as an individual human being, essentially, what makes you, you? This could be in relation to your cognitive capability, your values, and motivation, your skills and behaviors, and your strengths and preferences.

As such, using an AI assistant to answer these questions based on what is surmised to be the right answer, rather than the answer that represents you, clearly doesn’t help anyone.

Indeed, ChatGPT itself tells us this, when asked the question ‘How might candidates use ChatGPT to cheat on psychometric assessments?’:

“As an AI language model, it is not appropriate to provide information on how to cheat on psychometric assessments or any other unethical behavior. Cheating on psychometric assessments undermines the validity and reliability of the assessment and can lead to inaccurate results, which can have serious consequences for the individual and the organization that is using the assessment.

“It is important for candidates to understand that psychometric assessments are designed to measure specific traits, abilities, or competencies that are relevant to the job or task at hand. Cheating on these assessments can lead to a mismatch between the candidate’s actual abilities and the requirements of the job, which can ultimately lead to job dissatisfaction and poor performance.

“Furthermore, most psychometric assessments are designed to detect and prevent cheating through various measures, such as time limits, randomized questions, and validity scales. Attempting to cheat on these assessments can be detected and may result in disqualification from the selection process.

“Instead of trying to cheat, candidates should focus on preparing themselves for the assessment by familiarizing themselves with the types of questions that may be asked, practicing their skills and abilities, and being honest in their responses. This will not only increase their chances of success in the assessment but also help them find a job that is a good match for their skills and abilities.”

Nonetheless, in a competitive application environment, it’s likely that some candidates at least will try and use AI assistants to give themselves an unfair, unethical, and inappropriate advantage. To this end, talent assessment needs to stand ready to prevent, deter and detect cheating wherever possible.

To support this work, the Cappfinity Cheating Squad, comprising psychologists, data scientists, and software engineers, have been working for a number of years to review, test and implement a host of different anti-cheating measures across the diversity of our talent assessment suite.

These include solutions across each of the four pillars of the Cappfinity Multi-layer Defence against cheating, which is constructed from the interlocking, interwoven actions across Technology Solutions, Assessment Design, Scoring and Data, and Human Oversight. Our work in this area, across 22 different interventions to date, has been reported exclusively and confidentially to Cappfinity clients.

In reviewing each of these 22 different interventions, we consistently saw patterns and trade-offs emerge, which led us to conclude that any cheating intervention must be about finding the right balance between these trade-offs.

The biggest trade-offs we saw, time and again, were about diversity and inclusion, and financial cost. The cost of doing something is what it is, a simply calculated, rational, economic decision. But the trade-offs for diversity and inclusion gave us a lot more pause for thought.

Talent assessment has helped us make enormous advances in social justice over the last 10 years, increasing the representation of women, people from ethnic minority backgrounds, people from less advantaged backgrounds, and people with disabilities or neurodiversity. Talent assessment has done much to genuinely level the playing field of opportunity by allowing people from any background to be considered, on an equal basis, for some of the most desirable jobs available. 

In determining how we respond to cheating and the rise of generative AIs, we encourage every employer to reflect carefully and soberly on what matters to them, to ensure they find the right balance across the many competing priorities you will be juggling. Deploying the Cappfinity Multi-layer Defence against cheating gives you the best possible chance of achieving the outcomes you want.