As data scientists, a lot of people ask us if data science is better than human intuition. We’ve never been comfortable with the question and here’s why: it’s a false choice. Yet since this question comes up so often, let’s take a minute to flesh it out a bit.
For starters, non-data science businesspeople often look at relatively small samples, but in more depth. A retired CEO may have worked for 10 companies during her career (tiny sample), but she knows each story in rich detail. She knows the tangible facts (ex. financial outcomes) and intangible tidbits (ex. how a drunk Vice President nearly ruined everything at the holiday party).
Data scientists, on the other hand, tend to look at larger samples, but often in less depth. They may examine 10,000 corporate holiday parties to accurately predict when VPs will get drunk and cause a ruckus, but they may not know what soothing words were used to talk a specific drunkard down from the radio antenna.
Both perspectives are important. For example, knowing cases in greater depth ensures that quantitative research is grounded in real-world causality and is less likely to be spurious. Intuition can also help data scientists figure out what variables to consider and how to build an appropriate model.
Meanwhile there’s something to be said about statistics and larger samples. Nobody with a terminal illness wants to hear the doctor say “I’ve only given out this pill twice. Both patients died, but I knew each of them really well.” It would be a lot better to hear “this pill has been given out 10,000 times and only two people have ever died. I never met those two people, but there’s a 99.98% chance the pill will cure you.”
The best of both worlds is, of course, large samples that are understood in great depth. That’s the holy-grail.
Non-data science approaches also tend to be faster at first, but slower later on. For example, a lot of business decisions have to be made quickly. There isn’t time to build a predictive model or to even glance around for patterns. If your customer threatens to walk out the door unless you say “yes or no” in the next three seconds, you’d better say something… quick.
Relying on your wits is part of doing business. However if there are big problems that keep resurfacing, it’s a lot slower to go on guessing. If you don’t bring data science or some other form of rigor to the table you may never get a grip on what the underlying problem is. In the long term it’s a lot slower to treat the symptoms (keep guessing) than to cure the heart of the disease (ex. use data science).
In contrast, data science is often slower at first, but faster later on. It can take hours, months or even generations for data science to model, predict or solve some of business’s toughest riddles. However once you’ve built a robust tool, it’s relatively fast and easy to handle or even prevent those problems if they threaten to pop up again.
A final difference is the “intuitive-ness” of the answer. If you’re facing a problem where the answer is intuitive, then human intuition may be a good tool for solving it. Yet if you’re facing a problem where the answer may be counter-intuitive, data science is a good approach since human intuition is more likely to lead you astray.
While these are generalizations (exceptions certainly exist), data science and non-data-science-based approaches tend to differ in the following ways:
Seen this way, the guiding question becomes: what circumstances are you in? What kind of problem are you trying to solve?
If you need a quick decision for a highly intuitive problem that isn’t likely to come up again, a non-data-science-based approach may be good enough. However if you have a big problem that keeps coming up and the answer may be counter-intuitive, data science is probably the way to go if you’re willing to do some extra diligence up-front.
Perhaps the most exciting thing about intuition and data science is how well they complement each other. Like an old married couple, each one’s strengths complement the others’ weaknesses. They also help each other grow. Intuition can give data science a place to start looking for patterns and add depth to what patterns are revealed. Meanwhile the lessons learned through data science can become part of people’s intuitive understanding of the world and how it works.
So we beg you; resist temptation to pit one against the other. We’re not being diplomatic. The reality is, intuition and data science are good at different things. Like a shovel and a garden hose – you need them both, but for different tasks. It isn’t us versus them. It’s us plus them.