What makes them fascinating is the sheer number of applications for big data, even in a small market like New Zealand. Spark’s offshoot, Qrious, is using anonymised data taken from mobile phones, for example, to do all kinds of fascinating things, like working out who is attending ATEED events, how motorists respond to compromised routes, or where tourists going to the Hawkes Bay come from.

iStart got on the phone to Neil Mendelson, VP of Big Data Product Management at Oracle to get a perspective on just where we’re at with big data. Gartner’s Hype Cycle, which we find quite beautiful, provides a handy point at which to kick off, and Mendelson tackled it with enthusiasm. “At the beginning, we always see a tremendous amount of hype from vendors and analysts, but things tend to quickly descend into the trough of disillusionment as we find out that the goals initially imagined are a little more difficult. In time, that moves into the period of enlightenment, when things move into ‘suitably deployable’ status.”

That could describe any hyped up new technology (and there is no shortage of those). But Mendelson pressed on. “With big data, there are a number of differences which are emerging; from an economic point of view, we’re seeing the emergence of the digital economy, with big data and the internet of things in support; we’re seeing huge levels of disruption occurring not only at the points of origin [of the disruptors, many of which are in Silicon Valley] but in other parts of the world. Beyond that, we’re also seeing the incumbents rising up [and responding to the disruptors].”

What’s that got to do with big data, then? “What’s happening with disruption is that there is a lot less time which can be spend in the trough making mistakes. As time periods become compressed, there is an increased desire to take risks and endure failure with new business models. With basic infrastructure, many companies are look for safer ways to get over the trough; when the big data market started, there were startup companies with unlimited funds available to play around with technology. They could afford to start out by building their own cluster, the techies could experiment with abandon with components of the technology to see what was possible.”

By a cluster, he is referring to the grunty data centre stuff which is necessary to crunch high volume, high velocity data.

Good foundations
But Mendelson made the point that what impresses techies doesn’t always impress the money men. “Businesspeople care about results, so we’re starting to see enough companies stumble as they put together a cluster, which turns out to be a lot more difficult than they were led to believe and a lot more expensive. It also requires the right people and skills, and that’s starting to hurt the ability to expand.”

Pulling businesspeople out of the trough, said Mendelson, is the job of vendors which offer products that simplify things – and that means the cloud and preconfigured appliances.

“What we’ve focused on [at Oracle] is operationalising new technologies [like big data] within the context of what we have in the existing enterprise. That means more automation and less complexity.” In turn, it means the ability to cut down on the time taken to ‘implement these things’, he added. “Where getting a new cluster up and running once took 6 to 8 months, we’re able to shrink that to a matter of weeks or days through automation and pretuning of the complex subsystems.”

If that sounds amazing, Mendelson is quick to beat down any embellishment. “It’s not rocket science,” he said.

But, then, where does he then peg it all on the hype cycle? “Well, the hype is still there, but we’re seeing a transition taking place. We are focused on outcomes rather than hype and we are a product company. We’re still seeing companies take this on themselves [the infrastructure component of big data] and they will fall into the trough, but there is increased interest in packaged solutions.”

Pointing out that time is money in what he calls a ‘hypercompetitive market’, Mendelson said those looking to deliver practical big data solutions are welcome to have a crack at sorting the infrastructure themselves, but must accept the risk which comes along with it. “Premade solutions means high levels of automation. It also means getting infrastructure at a price point you cannot achieve on your own. We’ve recognised this as a commodity market with a strong focus on the cost equation and time to value. That might not interest technologists, but it should interest those looking to get value out of big data initiatives.”

…