Published on the 06/08/2014 | Written by Newsdesk

While a nation’s share of all the data on the planet is a very rough yardstick by which to measure national prosperity and track geopolitics, it does provide a clue as to the challenge that local businesses may face when attempting to gain global traction by exploiting their data reserves.

The Digital Universe study, conducted by IDC at the behest of EMC, found that last year 60 percent of the digital universe was controlled by organisations and individuals based in mature and developed markets. By 2020 that same proportion will be controlled by the emerging nations such as Brazil, Russia, India, China and Mexico.

Although developed nations’ share of the digital universe may decline, the volumes of data under management will do anything but. 

The report says that in 2014 A/NZ businesses will have 144 exabytes of stored data. By 2020 that will balloon to 874 exabytes.

According to EMC’s chief technology officer for A/NZ, Matthew Zwolenski, enterprise systems architects need to take heed of the trends and design systems able to cope with the 60 percent per year data growth rates currently being experienced across the region. He said that it was important to design systems not to cope just with current demands, but to be able to meet future requirements.

“If the system I’m building today is for one petabyte, it will need to cope with two petabytes in two years’ time,” he said.

Business should take comfort however in the fact that IDC said that only 1.6 percent of that data is “high value”.

But how to identify the wheat from the chaff?
Zwolenski said that 23 percent of organisations’ data was generally deemed to have value, with 1.6 percent having high value, but he said that while organisations could generally identify the valuable data, it could be difficult to work with as it could be stored in multiple silos spread across an organisation. He recommended enterprises create systems which allow “data lakes” to be established so that more of the really valuable data could be easily found and analysed.

Gartner however this week issued a caution about these so-called data lakes. “In broad terms, data lakes are marketed as enterprise-wide data management platforms for analysing disparate sources of data in its native format,” said Nick Heudecker, research director at Gartner. “The idea is simple: instead of placing data in a purpose-built data store, you move it into a data lake in its original format. This eliminates the upfront costs of data ingestion, like transformation. Once data is placed into the lake, it’s available for analysis by everyone in the organisation.” 

But Heudecker warned that getting data into the lake was only one part of the challenge. Companies still had to manage the organisation of data in the lake, properly secure and protect it, ensure it was accessible in a timely manner and that appropriate information governance rules determined its use. For some companies a more traditional and ordered data warehouse might still be a better option he warned.

Lake or warehouse, there will still be debate about what data goes where. At present just two percent of A/NZ data emerges from embedded devices, the so called internet of things, but by 2020 that will soar to 12 percent according to IDC.

Analysing this internet of things for insights about organisational activity and consumer demand has been a talisman for the big data movement.

However a separate report issued by HP has warned that 70 percent of devices currently used to create internet-of-things networks feature vulnerabilities in terms of privacy settings, authorisation and encryption. A full 60 percent of the devices did not encrypt software downloads, rendering the device particularly vulnerable.

If EMC’s forecast is right and 12 percent of all data collected by 2020 will come from such devices, it will be critical that these vulnerabilities are addressed urgently in order not to compromise organisational data integrity.