Today we got a new intern (lets call him Tom), and while my colleague (that would be Jane) was trying to find a suitable task for him to get started with, I overheard a conversation similar to the following:
Tom: Uhm, not sure. I’m not even sure what the numbers mean in this context.
Jane: Well, it goes from “I never heard of it” to “I know everything there is to know”.
Tom: I really can’t say… Maybe a five? …
$foo on a scale from
Let’s take another look at why this or any other arbitrary scale is no good for skill (self-)assessment. We could assume that Tom was brought up in Spain, where the school system rates you from 1 (worst) to 10 (best). While in theory this could be a linear scale, in practice it is far from it: 1-4 means that you’ve failed the class entirely and have to repeat it. The remainder is mapped as follows:
- 5 ≡ sufficient
- 6 ≡ good
- 7-8 ≡ notable
- 9-10 ≡ outstanding
So now I’ve critized the one-to-ten-scale. “But which one should I use instead?”, you ask. The answer could be: whichever you like, just keep the scale to yourself.
Using binary search to position someone on a scale of your choice
log2(10) steps (3.32, for the 1-10 scale), by using binary search starting from the middle of the scale.
(1 2 3 4 5 6 7 8 9 10) - Pivot: 5. Answer: Correct ( 6 7 8 9 10) - Pivot: 8. Answer: Correct ( 9 10) - Pivot: 9. Answer: Incorrect = Result: An 8
Since a scale is never communicated to the interviewee, no misunderstandings regarding the nature of the scale can arise. An additional benefit of this technique is that you can refine your questionnaire over time, based on your observations, thus gradually increasing the precision of your assessments. It should be noted that these must not be questions that can be memorized but could also be small programming excercises or the like, thus making this a viable technique for programming interviews as well. And it can even be automated!
What do you think about the one-to-ten-scale? Is it useful, and if so, in which situations? Could binary-search-based assessment be next big thing™?