Great tidbit from Elon Musk at the Ignition Conference on how having job applicants explain their thinking at multiple levels helps him figure out if they really worked on the problem.
If you just talk to the people on your team you can learn a tremendous amount. As you iterate through problems … when you struggle with a problem that’s when you understand it. Once you’ve done that for (years), then you have a pretty good grasp of it. In fact, that’s one of the ways, when I interview someone … is to ask them to tell me about the problems they worked on and how they solved them. And if someone was really the person that solved it, they will be able to answer at multiple levels — they will be able to go down to the brass tacks. And if they weren’t, they’ll get stuck. And then you can say, “oh this person was not really the person who solved it because anyone who struggles hard with a problem never forgets it.”
This connects for me a bit with the false record effect:
A group of managers of identical (moderate) ability will show considerable variation in their performance records in the short run. Some will be found at one end of the distribution and will be viewed as outstanding; others will be at the other end and will be viewed as ineffective. The longer a manager stays in a job, the less the probable difference between the observed record of performance and actual ability. Time on the job increased the expected sample of observations, reduced expected sampling error, and thus reduced the chance that the manager (of moderate ability) will either be promoted or exit.
It’s also a good trick for the batesian mimicry problem — that is, separating those who know from those who act like they know.