Published on the 28/09/2016 | Written by Donovan Jackson

Insights so good they might freak you out…

In today’s heady environment, which some have called the most exciting time to be in IT since that whole Year 2000 thing, the potential power of data is coming under the microscope with the concepts such as big data, machine learning and predictive analytics quickly entering the lexicon of business.

And such is the power of information analytics today that it can even become a little uncomfortable, said Donald Farmer, principal at TreeHive Strategy and former Qlik VP of innovation and design, particularly emerging ‘just in time’ approaches which can provide insights so surreal as to be slightly disturbing. “It’s an uncanny valley situation; there comes a point that something robotic approaches humanness so closely that it creeps you out. Then you go beyond that point and get into Blade Runner territory, where you’re OK with it again.”

He said effective use of data analytics has to get to that point, where it is not creepy or uncanny. “‘Too insightful’ can be weird – but, of course, that point differs for different individuals.”

The target moves, too, because that most common example of machine learning, Google and more specifically Google Now, might once have occasioned a mild sense of paranoia with its accurate predictions. Some years after its introduction, there’s nothing remarkable about a service which knows where you are and what you’re doing.

While Farmer said those in business tend to have a higher ‘uncanny valley’ point, data specialist and WhereScape CEO Michael Whitehead said in his experience, local businesses are also a little worried about the implications which come with the ability to access powerful data insights. “The amount of data out there and its consolidation in the hands of a Facebook, or a Google or indeed the government – there are downsides to that and there is the risk of unwitting disclosure which can have unintended and unpleasant consequences.”

Whitehead said the problem with data, especially private data, is how to handle it and how to make it private enough. “It has to be open enough to be useful, but it has to be private enough to prevent misuse.”

Farmer said trips into uncanny valley could be expected in the emerging field of ‘just in time’ data analysis, where streams are taken directly from line-of-business systems or other sources (like IoT sensors) where there isn’t any obvious opportunity for metaphorical intervention such as classification and categorisation – the traditional ways of ‘doing BI’. “There’s also far less structure applied; the data essentially undefined right up to the point where you want to make use of it and apply meaning to it.”

Compare that, he said, with a data warehouse. “That data is a record, a metaphor for something which has happened: it records a sale, but isn’t a sale. It records a customer, but that isn’t the customer.”

Stored data, unlike the live stuff, also becomes very malleable. “It can be classified and reclassified. It’s like changing the category of a book at will; a Lance Armstrong biography, for example, could be categorised in sports, might once have been classified in autobiographies and now find itself in fiction.”

To take better advantage of data today, Farmer said IT departments need to go ‘from gatekeeper to shopkeeper’. “The traditional view of IT is that it is there to stop bad things happening. That results in a one size fits all approach to governing the data system which, in turn, results in bypassing of the department and shadow IT. Instead, the IT department should say ‘we have this information that we’d like you to use and we will make it right for your purpose’. That change in approach is necessary if you want to maximise use of data within your organisation.”

Farmer and Whitehead are presenting at the Masters of Data event, taking place in Auckland and Wellington on 8 and 10 November respectively.