As a venture capitalist/data scientist, I’ve spent over a decade arguing that other venture capitalists and innovators can improve their decision making (a lot) through data science. While the idea seems like a no-brainer today, 11 years ago it was heresy.
Reactions to my work were historically hostile and emotional, and that’s putting it politely. Words like “data science” and “big data” hadn’t happened yet, and several times a day I’d be accused of trying to use “algorithms” to “kill creativity”. Innovation was synonymous with creativity, serendipity and sparks of human inspiration. Algorithms were synonymous with cold, naïve, blind faith in numbers – the stuff of ivory tower geeks and wonks who didn’t know how the real world worked. Smug people would say catch phrases like “garbage in, garbage out” as if it was some kind of debate-ending checkmate.
Thank goodness that phase is over, at least for the most part. Instead of being attacked daily, now months can go by without anyone feeling upset by the basic premise that data science can help innovators identify opportunities and navigate uncertainty. The idea has finally become mainstream, feeling (to some people) like it had always been there. A few more quant-oriented venture capital funds have popped up, platforms like Crunchbase and Pitchbook are making startup data more familiar, and most big VCs have begun “playing with data,” albeit with varying levels of sincerity. That said, I do still get the occasional snarky comment; now almost exclusively from venture capitalist holdouts.
This has made my life a lot easier. Better conversations now happen more quickly, without as much distraction and preliminary heavy lifting around “what is data science” …and “yes, data and math really can be helpful,” …and “no, I’m not trying to kill creativity or replace humans with robots.” I no longer have to use examples of how data science has helped other industries (ex. Moneyball, healthcare, manufacturing), and when I say things like “machine learning,” people’s eyes don’t instantly glaze over.
This is progress, to be sure, and now people are beginning to learn the differences between better and worse data science. Not all data science or “algorithms” are created equal. This can be daunting because helping someone understand key differences between various approaches to data science is more nuanced than trying to get them to understand what data science is in the first place. Think of it this way – I don’t know what makes one Poodle lose a dog show while another wins. Yet at least I know what a Poodle is, and I’m unlikely to confuse it with a Basset Hound. That may not be much, but it’s a start.
The need to guide innovation decisions with data science gets more acute every day. Top threats and opportunities are increasingly spread like dust across ever-widening geographies. The best technology deals used to be concentrated in Silicon Valley, but now they’re appearing in random cities all over the world. This creates a logistical nightmare for investors or companies who still rely on human labor for insight or deal flow. You’d almost have to hire an analyst in every city on Earth to watch patiently in the hopes that one day, something important happens there. Relying on deep domain experts is also getting less efficient every day as industry categories blur and focused specialists see proportionally smaller slices of the whole picture.
Innovation has traditionally been labor-intensive, which simply hasn’t kept pace with today’s global, domain-crossing, rapidly changing markets. These problems beg for analytic solutions, or at least technological ones, without legacy constraints imposed by pure human labor. Instead, people need an ability to sit in one place and have the world brought to their fingertips in meaningful ways. They heed to augment their efforts with technologies that increase their scope, speed and precision. A data science arms race is happening in the world of innovation decision making, and those who’ve been slow to recognize it become less competitive by the day. Faced with the realities of today’s innovation landscape, analytics have crossed the tipping point from “nice to have” to a “must have.”
The fight to bring data-centric tools to innovation decision making has enjoyed major steps forward during the past couple years. It’s come a long way and I – for one – am grateful for those of you who are here for the journey.