C&B Notes

Making Sense of Big Data

‘Big data’ is transforming how decisions get made in a variety of industries, with the underlying thesis that better choices are made when decisions are driven by data rather than by intuition.  This shift is generally positive, but big data is not a panacea.  Its effectiveness is ultimately determined by the people interpreting the information.

Messy, ambiguous data and hidden biases underscore a growing need to hire and train data vigilantes to watch over and ask “why?” about our every interpretation from big data.  Big data kitsch promotes a world of blissful ignorance in its focus on correlation without explanation.  But the data vigilantes do need to understand “why”, sometimes to debug a spurious correlation or systemic failure (like we saw with Google Flu Trends), and other times to be able to develop a smarter method to measure the thing that we really want to measure.

It can be tempting to use data as a crutch in decision-making: “The data says so!”  But sometimes the data lets us down and that exciting correlation you found is just a by-product of a messy, biased sample.  More advanced algorithms can sometimes help cut through the mess and correct the sample, and smart skeptics can help step back, reflect, and ask if what the data is “saying” actually fits with what you know and expect about the world.  Hiring and training these data vigilantes as well as inculcating a healthy dose of data skepticism throughout your culture and team can only help bolster the quality of decisions you ultimately make.