What is Survivor Bias? It is a bias in selection. You may be misled by what you optimistically think leads to success due to having a limited view of all factors. Effectively you fail to see or ignore failures along the way. But how do you overcome this?

The traditional example is from World War II with planes that returned from raids. Initial analysis recommended that armor be added to the areas on returned planes that showed the most damage. This was a false assumption as the returned planes could sustain said damage. The bombers that had actually been shot down were not present for the damage assessment. Thus, reinforcing the areas where the returning aircraft were unscathed represented the truly vulnerable parts of the plane.

survivor_bias.png

Why is this important? Well, it highlights that we often tend to attribute success to superficial signals rather than understanding the context that led to an entity’s success. When hiring, what is your sample size and control group for evaluating an individual contributor’s claimed skills, experience, and successes? How can they prove their track record is due to their own capability and not solely due to external or other organisational factors? Can you rely on a candidate to deliver results if hired, or was their previous success due to other team members, embedded processes, or just timing? So the question then arises, what capabilities can you independently and actually verify?

Risk and Culture

Culture is constantly being patched in an organisation. Employees morph due to their experiences and environments both in and out of work, yet when a new hire comes onboard and merges with a team, culture shifts once again (whether for better or worse). No matter how hard you try, you can not outsource your organisation’s risk when patching your own culture. Albeit some will say you can transfer risk, your organisation still ultimately owns its own outcomes. Irrespective of finger pointing or a single throat to choke, risk can not be transferred. All you can do is mitigate, minimise, or accept risks. When confidence is low in internal talent sourcing and screening, some companies place a premium on another company’s brand (and their hiring program). They then consciously bias applicants from those companies. This can be seen as an attempt to acquire pre-vetted “talent” from other organisations. This shortcut is a dubious stopgap rather than creating better internal sourcing and screening processes, processes that should independently and objectively verify candidate skills.

Brand Based Acquihires

This brand “acquihire” practice can be thought of as trying to acquire a slice of another company’s success, culture, or trajectory. It is a brand based filter and an attempt to simplify decision making via a pre-qualified talent pool (oftentimes existing as an unofficial policy or open secret in organisations). In closed circles you may hear “I want someone from Facebook, Apple, or Google”. This can also lead to those new hires subsequently justifying technical or business decisions with “this is what we did at company X” (which alone, is never a sound approach!). There is an assumption that person Y’s experience with (or simply exposure to) a technology or scale, will somehow directly translate to success in your organisation. There’s also an underlying expectation that these new brand hires will be able to de-risk your organisation’s trajectory in to the future…

Unfortunately CVs and Résumés are fundamentally broken thus the risk still falls on your organisation’s hiring team to verify a candidate’s skills to ensure they will be effective and at least, hit the ground running.

dirty_lens.jpeg

Peter and Dunning-Kruger in Action

Typically you may never really know why someone leaves their previous organisation (as you are only privy to what they choose to tell you!).

  • have they reached a ceiling and are now seeking new challenges?
  • have other life or family changes initiated a move?
  • is it a financial decision, better remuneration, or for increased flexibility?
  • are they on a PIP(Performance Improvement Plan) and already being managed out?
  • or more scarily perhaps, have they already reached their level of incompetence ( e.g. the Peter Principle) ?

There could be lots of reasons but for any hire, it’s still incredibly important to distinguish between a candidate’s confidence and competence. Where is their proof of experience and can they demonstrate their competence?

The Dunning–Kruger effect is a cognitive bias wherein people of low ability have illusory superiority. They mistakenly assess their own cognitive ability as greater than it really is. Additionally, there is a pronounced difference by traditional genders, as noted by The Atlantic in a piece called “The Confidence Gap” which notes;

In studies, men overestimate their abilities and performance, and women underestimate both.

So, how does Dunning–Kruger affect self-written CVs and Résumés? Also, what actually happens in early interviews or screening activities if the person screening a candidate or CV is not proficient in the domain themselves? At what point should you ask candidates to demonstrate their claimed skills (and additionally how, when, and where)? Even if you are hiring solely for attitude (rather than experience), can a candidate hit the ground running or will they require months of specific training to become competent and effective?

Caveat Emptor

It is extremely time consuming for domain experts or senior staff to independently screen and verify all applicant or candidate skills. This checking of fundamental hard skills can however be automated in many domains. This saves on expensive human time and speeds up time-to-hire whilst also facilitating wider recruiting funnels. Additionally, imagine having your own candidate skills analytics to use in your structured hiring process to support not just decision making but as talking points in later stage interviews with your senior engineering staff or leadership.

PS
At PanSift we recommend that organisations request automated initial technical screens. By using a position description, CV, or candidate profile as a form of keyword currency, we then use ML (Machine Learning) and NLP (Natural Language Processing) to extract testable skills. You can also use predefined simulations or even drag and drop skills keywords to automagically build your own. We call the process of testing candidates in real live scenarios "Simulation Based Screening". It allows for wider funnels and less bias whilst saving costs and reducing time-to-hire.