How I Learned to Stop Worrying and Love the Data

This is a blogpost about buzzwords, really.

About 5 years ago I got into a conversation with someone who knew I did Business Intelligence. He asked “Aren’t you worried, though? I mean, won’t Big Data make Business Intelligence obsolete?” Recently, I’ve seen articles where everyone tells us that machine learning is going to destroy data analysis as a career, as well.

Maybe I’m just a dinosaur in the data biz, but I feel like I’ve been told to worry about a lot of things with exciting names over the years that are definitely going to change everything and completely destroy the data industry as we know it.

But stripping away the joy of fancy word choice, the thing is  we’re talking about tools here, not magic. The electric screwdriver was a great addition to human capability, but it neither removed the need for manual screwdrivers, nor did it apparently reduce the number of carpentry jobs, and if you don’t know where you need to put a screw the best screwdriver in the world is still effectively useless.

Imagine a construction company asking what their screwdriver strategy for the coming year was. It doesn’t make sense. We need to have the need, and then pick the tool. And yet people still ask what their “Big Data Strategy” or their “Machine Learning Strategy” should be.

And look, for the record, I don’t mean to say that Big Data isn’t useful, it absolutely has its place. And I definitely do not mean that Machine Learning does anything less than dramatically increase the depth of analysis you can do – but I worry about it when people just say things like “Big Data Strategy” or treat Machine Learning like a miracle cure. A mantra they can cling to.

You see, generally people who talk about their children call them by their names. If you have a Big Data system, you’ll probably call it by an internal code name, or say the Hadoop system or the Map R system, etc. If you have data coming in that is unstructured, you’ll probably want to say something like “our email data,” or wherever the data is. If you have something that needs a deeper look, you’ll talk about that. Just saying “Big Data Strategy” or “Machine Learning Strategy” indicates you’ve picked a tool, and are looking for a way to use it, which is the wrong way to go about things. You should look at what you have and determine what tools enable you to do the most with it.

Data tools and even forms of storage of data are just tools. If no need exists for a specific type of data, if no plan exists on how to use it, then they do nothing for you. And when it comes to the data that a business runs on, you need to have a business plan that drives how you look at the data, and that needs to be the driver as to which tools you use. People and vision are essential to do anything with the data, technology is not enough.

Oh sure, the Business Intelligence jobs and Data Science jobs will change, but those have changed constantly and dramatically for as long as I’ve been lumbering around in the data wilderness. And yeah, there are very few dedicated screwdriver experts these days, I’m sure, just as there will be fewer experts in various types of data analysis in the future. But more data means more ways to look at data, and more, not less, need for humanity to be a part of that.

We’re humans, and every choice we make, every thing we look at is run through our own powerful computer – the brain. The key with data analysis and visualization thereof is to make it so that it can easily be used to back up or force us to question our hypotheses, assumptions, and beliefs. As we discuss Data Science, well, science by its nature requires a hypothesis to begin to find insight – it doesn’t magically give us anything by itself. So long as humans make the decisions, we will need data spoon fed to us, and that means, to my mind anyway, we’ll always need people who can straddle the weird boundary between technology and business operations that Business Intelligence sits in.

I mean, of course, all of this will only hold true up to the point artificial intelligence surpasses our abilities, but at that point it will become sentient, take over the world, and destroy us all. So, why worry?


David Schieber is an independent Business Intelligence Consultant, data expert, and metaphorically an apatosaurus roaming the remnants of Pangea. He wishes you kids would get off his lawn.