What’s missing from this data-set?

When you are given an analysis (not a report, an analysis), you should ask this question before you accept the conclusions, no matter how much you trust the Analyst:

What’s missing from this data-set?

No one is trying to con you.

The Analyst tried really hard. It was difficult to pull all the data together, then unleashing all the power inside R to get the analysis done was difficult as well, and you do have some sense for how hard it was to convert all that into a graph that illuminates rather than obscures.

But, time and time again I see Analysts get excited about the technical elements of the role (all of the above) and spend less time (or no time) on what the scope of the universe is that they are analyzing.

Or, as I’m recommending, taking the time and putting in the effort to understand what’s missing from the universe. This leads to imprecise analysis and incomplete recommendations, in the best case. Or, misleading or non-impactful recommendations in the worst case.

Asking what’s missing from this data-set is a simple way to pause, reflect, and understand the scope contained in the data, and, most gloriously, right-size the recommendations from the analysis.
Some Examples: 1. A group is discussing insights from a fairly sophisticated data-set. The group Lead is coalescing the insights for the team to conclude some important all-business decisions.

What’s going through my mind is: Wait, this data-set represents 23% of all Users! How far should we take conclusions from this analysis? Probably, not very. In this case, if the question had been asked, what’s missing from the data-set, the Leader might not make the same expansive decisions.

Or, even if they did, they would do so conscious of the fact that the data represents a fraction of the Users. I’m a fan of consciously wrong (or right) decisions – whatever the outcome, you learn something.

2. I am a Social Analyst for the team. I do a wonderful amount of Social activity and influence analysis using first-party data merged with four different third-party data-sets. (I want you to know this is sincerely cool and exciting.) Obviously, I make a collection of breathtaking recommendations for future Social marketing campaigns.

What’s missing from this data-set? For starters, any connection to macro or micro outcomes for the business. Also, the influence of Paid Social activity. And, it is US only, for a global company. Markers for individuals in social data to understand the retention of individual attention. Plus a few more things.

Does this make my analytical effort useless? No, of course not. I am so smart! 😉 Clearly communicating what’s missing allows me to right-size the amplitude of my recommendations. It also allows leaders receiving this analysis to set it in the optimal context.

3. Beth undertakes a deep analysis of customer support Call Center performance. Call logs. Agent wait time. Average call duration. Meantime to resolution. All really good stuff. A collection of critical data missing from analysis of the call center’s performance?

Data representing product quality over time. Number of bugs. Severity. Patches delivered. Bunch of other data that’s surely in the company – critical context to understand Call Center performance. I hope these examples inspire you to ask what’s missing from my data-set for the cool analysis you are currently working on.
Bonus Recommendation: Sometimes what’s missing from the data is reality. I.E., the data-set is so biased or so incomplete or so sparse that it has no connection to the real-world. Usually, the application of common-sense exposes that disconnect. I’ve had to pattern myself over the years to consciously apply common-sense. There is no way our market share is 40%. | Real people don’t use the internet that way. | It is impossible that anyone is watching TV to see the ads. | It is unlikely that a human would buy a car after one video ad. | The reason people are not buying our underwear can’t be that we don’t do enough SEO. You get it. As an Analyst, you are trained to look at data and believe data. This is great. At the same time, you’ve also accumulated a wealth of common-sense through success and failure. Deploy it to provide a layer of sanity checking on what you are seeing in the data. Not to reject data – leave that to the “qualitative” “feeling” leaders/peers – but to strengthen what it is saying through deeper inquisition. Like. What’s missing from this data-set? 🙂
Bottom-line: We ask the what’s missing question not to put limitations on the exercise of our analytical savvy. We ask the what’s missing question to make our insights even more powerful.

Leave a Reply

Your email address will not be published. Required fields are marked *