A few weeks ago, we were in discussion with a client about them considering using our mental toughness questionnaire – the MTQPlus – in their development programmes. The MTQPlus is a well-researched and very well-evidenced measure.
The client seemed interested but noted, “I already have a measure that might do what yours does”.
So, I asked and was told what they were using for selection and for development. No names but it was an ipsative measure that would not be very useful if you are colour blind.
They added, “It’s Jungian! It’s a psychometric measure, isn’t it?”.
So, I asked what does that mean? “We are one of 4 types”, they added,“ That’s what the training covered”.
Next, ”What about its reliability?”.
“It’s good – it works every time”.
Of course, that is not what I meant. I explained what reliability was and why it was important. There was no response.
“What about validity? How do you know it assesses what it claims to assess? And indeed, are you assessing something relevant to the task?”.
Again, silence.
Now we have an uncomfortable situation.
I am not concerned that they are using a particular measure. Some will argue that even measures with low reliability can start a useful discussion.
But a good psychometric, and there are many, can do a great deal more than this. They are incredibly powerful tools in the hands of someone who knows what they are doing.
That is key! You need to know what you are doing. A psychometric measure is a complex and technical instrument. All measures are not the same. They all do different jobs. Some are better than others. You do need to know how to tell them apart.
This incident is not unique. We come across this a lot of the time. People attend a workshop for a product and think, “That is it. I know enough”. Scarily, some declare themselves as an “expert”.
In this case, the CEO had attended a workshop where this measure was used and had decided that is what would work in their organisation. It is not the best way of bringing knowledge and skills into an organisation.
There are sources of guidance for psychometrics. The British Psychological Society accredits well-designed Test User training programmes which introduce what you need to know.
We have launched a simple, inexpensive training programme that guides you through what you need to know. Understanding Psychometrics for Coaches and Mentors.
I would not trust a surgeon to operate if he decided to use a butter knife instead of a scalpel because “It’s a knife, isn’t it?”.
Click here for a useful checklist for selecting psychometrics.
