Tom Davenport: big data is too important to be left to the 'quants'

One of the challenges with big data — or with most data analytics efforts over the years — is that it has typically been relegated to analysts and holders of Ph.Ds in mathematics or statistics. Thus, it's been highly intimidating territory for most business decision-makers. Organizations are striving to compete on analytics, and hiding the processes and management of data analysis in a back room isn't going to cut it anymore.

That's the view of Tom Davenport, visiting professor at Harvard University and co-author of the seminal work Competing on Analytics: The New Science of Winning. I recently had the opportunity to chat with Davenport, and the Q&A is posted at CBS interactive's SmartPlanet site.

Davenport, who is also a senior advisor to Deloitte Analytics, spoke about the difficulties of converting to an analytics-driven culture. He is finalizing a new book, Keeping Up with the Quants: Your Guide to Understanding and Using Analytics, which makes the case for better communicating analytics to business decision-makers.  

We touched upon a number of topics, and I was curious why he felt that the existing BI and analytics tools that have been on the market — graphic tools, dashboards, balanced scorecards, and the like — weren't up to the task. Davenport sais that it's time for a new type of platform for communicating data analytics results — a more "narrative" approach.

"We've all grown up on pie charts and bar charts, but there are probably at least tens, if not hundreds of alternative approaches to visual analytics," he explained. "Narratives are a pretty good way to convey information in the past, so maybe we should be converting our data and analysis into stories. People are starting to do that more. Most analysts were unfortunately not trained in how you communicate effectively about analytics, so we've got a long way to go in terms of doing a better job of that."

I also asked Davenport about the role of cloud platforms in promoting greater availability of analytics, and he said cloud isn't quite there yet.  Here's what he had to say: "It's certainly making a difference in terms of analytics on big data because its almost infinitely expandable, much cheaper, people don't have to build up the capability to analyze huge amounts of data all the time. They can expand to suit their needs. There are cloud-based versions of most of the visual analytics offerings, but they don't tend to be [as] sophisticated yet as the on-premises ones, so I think its not really driving things yet. For basic big data processing, and computation and analysis, visual stuff, narrative stuff — not as much at this point."

There is also a tendency to move or embed analytics into applications, with end-users fairly oblivious to what types of algorithms are spitting out the answers being acted on by applications. Isn't automation making it less necessary to understand the machinations of analytics? Too much reliance on automation may be a dangerous thing, Davenport responded. "We saw the challenge in financial services; you have situations like the flash crash, where there were all these automated trading things happening, and we had no idea why. The real challenge is going to be being able to trace the logic and how the algorithms work when things go wrong, so we can intervene and override."