Traditional businesses have learned a lot from hyperscale operators like Google, Amazon, Facebook and Twitter. They’ve learned the power of cloud infrastructure, of agile ways of working, and of open source tools that redefine how businesses procure and treat technology. They’ve also seen the power of good data as well as of good – and ugly – sides of data governance and custodianship.

These lessons are particularly important as data’s importance grows.

Data is at the heart of the new digital economy, a source of major upheaval for some traditional businesses, and, for better or for worse, central to most companies plans. It is already the source of innumerable market discoveries and innovations – and if it is to power the next wave, it is important that occurs ethically and in a way that does not repeat past mistakes.

Today, data is a key input to many business models.

Facebook, Google and Twitter remunerate themselves thanks to the data collected from users to whom they can target advertising.  Airbnb and Uber operate on data fuelled by their communities. Spotify and Netflix offer streaming and near real-time playback of personalised content based on harnessing mountains of data.

These companies were built on the exploitation of data. It’s at the heart of their businesses. Their success shows more traditional companies how data can help them win.

Data is becoming central to the operation of many business departments, from marketing and finance through to sales. Whether the data is in the cloud or in third-party applications, companies require it in order to develop new products and services.

From pizza delivery to insurance – all services can benefit and be made better through the integration and analysis of data sources, if companies are savvy enough to bring together data from a range of sources at their disposal and remain on the cutting edge in a technology environment that’s constantly evolving.

Data is also fueling technology company valuations. The acquisition of Mulesoft, a specialist in API integration, by Salesforce, the leader in CRM in the cloud, for $6.2 billion, almost doubled its share price in early February, demonstrates that the market alone is still full of promise.

Today, the need of companies to capture the value of their data, is driving the growth of the data tools market, including big data and business intelligence solutions.

Last month, IDC estimated that big data and analytics spending in Asia Pacific (excluding Japan) would top US$14.7 billion this year, growing to US$22.2 billion by 2021. Of that, Australia accounts for about 18 percent.

Globally, IDC expects big data and BI spending will be worth as much as US$210 billion in 2020.

Data is extremely important to technological innovations like artificial intelligence, which simply could not exist and develop if it were not fed by astounding volumes of data.

In addition, the autonomous vehicles that we are beginning to see on our roads are only possible thanks to our relatively new ability to process, in real time, all the data that the on-board computer produces from sensors determining everything from speed and position on the road, to the musical tastes of its owner.

In the future, data will power the evolution of quantum computers, the rise of brain-computer interfaces, and the discovery of preventative disease detection solutions. In order to get there, however, we need to have an open discussion about the ethics and merits of the data-driven decisions being made.

Recent events in the world have kick-started this process. Data can bury companies in bad press when it is misused, as the Facebook/Cambridge Analytica scandal proves. Its exploitation can have an impact on real life – in this instance, influencing election results.

New regulations have emerged in multiple jurisdictions that favour better data protection, notably: GDPR and e-privacy in Europe; the Privacy Act in Australia; the Act on the Protection of Personal Information (APPI) in Japan, and the Federal Law on Protection of Personal Data Held by Individuals in Mexico.

These sets of rules can be used as the basis to create a healthier environment, where the economic world rebuilds the trust of the users, customers and employees with whom it interacts.

The data leak linked to the Cambridge Analytica case is also pushing forward political discussions in the United States, a country known for its reluctance to regulate, to strengthen rights over the use of data.

Beyond regulation, discussions around ethics and values are also beginning to make their way into the debate around data and its use.

Such is the case with sensitive health-related data – a debate which Australia is about to have, as it looks to harness national e-health record data for ‘secondary’ uses such as medical research.

While this data could help save lives, the same data could also be used by a company to determine if a future client is likely to develop a disease and therefore become the basis of a refusal to insure the individual as a preventive measure. Who gets access to valuable data, in what format, and for what reason are all questions that need to be answered.

What is important though is that the challenges are worked through so that the full benefits of data can be realised. The economic world of today and tomorrow is – and will continue to be – structured around data.  At the same time, consumers and users are becoming aware of the value of that data.

A virtuous circle is taking shape, which will no doubt be at the origin of the next technological and societal revolutions; or at least that’s what the data points to.

ABOUT STEVE SINGER//

Steve Singer is ANZ Country Manager, Talend.

Traditionally, the collection, storage and processing of data has been a job for the IT department. With data centres full of servers, they’ve been data guardians since business computing first arrived.

Yet now, as data continues to explode in both volume and importance, it’s no longer good enough to gather huge amounts and expect it will be properly used. With data now underpinning business strategy, it needs to be in the hands of the right people, in the right form, at the right time. As a result, traditional roles within a business need to adapt, as CIOs and CTOs oversee comprehensive digital transformation projects.

The role of digital transformation
The aim of every digital transformation project is to create an adaptive, dynamic organisation. It aims for the perfect marriage of the business and IT function and requires both to collaborate to successfully harness all available data.

This is imperative to deliver the types of rapid growth and customer-centric developments that modern businesses are seeking to achieve. In recent years, groundwork has already been delivered through the increasing use of cloud platforms within businesses.

However, while the cloud provides the perfect platform for scalable, agile digitisation, three further challenges stand between organisations and true digital transformation success. These are:

1. The growing pace of business:
   Businesses in every sector are needing to operate faster than ever before while also dealing with an explosion in both data types and volumes. At the same time, consumers are expecting the very latest products, with personalised services, in real-time.The challenge comes from the fact that this data frequently ends up being siloed, out of context, or of bad quality. Indeed, industry estimates predict that working on flawed data costs a business in the region of 10 time more than working on perfect data.Traditionally, employees within a business have maintained this data, but this is no longer feasible in the face of the sheer volume of information being received. Instead, businesses need to be empowered by modern technologies such as Big Data and machine learning to ensure that as much data preparation, cleansing and analysis as possible automated.Being data-driven is a mandate for modern business, and the strain cannot be placed on IT to simply keep pace with the latest technological innovations. Instead, the business function must support the creation of a digital strategy, focused on the latest business objectives, for the company to succeed.

2. Digitisation is changing the job description:
   In the not-too-distant past, IT resources were centralised with an IT department managing on-premises data using legacy systems. While this was an effective way of keeping data safe and organised, it resulted in the data being hard to access and even harder to use.
   However, in today’s data-centric world, all employees must become self-sufficient when it comes to accessing and using data as a part of their daily role. The roles of everyone in a business, from the CIO to the business process analyst, are evolving so that they require data at their fingertips. Data is needed so they can strategize, execute and deliver for the business with the most relevant and up-to-date insights available.

3. Threats to data are increasing by the day:
   The knee-jerk reaction to this situation might be to make as much data as possible available to as many people as possible. However, any well-versed CIO knows, this is not viable.With upcoming regulations like the European GDPR and Australia’s data breach notification laws, organisations have an increasing obligation to make sure only the right people have access to every piece of information.Here the solution is successfully implemented self-service IT solutions, which automate functions such as data access requests and data preparation. This is fundamental to allowing business employees faster access to the right data, as well as providing clear lineages of who accessed what information and when.At the same time, automated data preparation tools are essential to reduce the burden on the IT team, performing manual cleansing and formatting tasks. This, in turn, will enable the team to focus on delivering new technologies for an organisation, rather than troubleshooting legacy issues.

The rise of the cloud has created the possibility for every person in every business to be data driven, however this state is yet to be reached. Instead, organisations are experiencing data silos and limits on innovation.

The key is creating an approach to data that is built with the business objectives in mind. By making data available when it’s required, where it’s required and in the form it’s required, these objectives can be met.

ABOUT STEVE SINGER//

Steve Singer is ANZ Country Manager, Talend.

Legacy vendors in the big data space generally have internal development organisations, dedicated to building proprietary, bespoke software. It’s an approach that has worked well over the years – but it is being supplanted by open source approaches.

That’s because the big data market has always moved fast and it’s had an element of open source from the beginning; for example, a large proportion of what the major Hadoop vendors deliver is based on open source. These vendors, and those building complementary technology, are best placed to take advantage of new big data trends (like Spark Streaming) and build solutions that add value to customers.

Traditional legacy and proprietary approaches to data integration still have their place.  These vendors have solid products, reliable technology and well-funded development teams. However, their products are typically built on a traditional architecture which may not easily adapt to the big data environment.

These products may work effectively for businesses that are doing things the way they always have. For straightforward requirements around data integration, data quality and ETL, while retaining existing processes and approach, there may be little need to move away from proprietary vendors entrenched in your information architecture.

The difficulty comes when organisations want to launch new projects or drive business transformation or product refreshes. Such moments can bring concerns around cost and flexibility.

For example, looking for additional functionality around big data ingestion could see legacy vendor licensing approaches becoming an issue. Buying perpetual software means major upfront costs and once the decision is made, it can be hard to modify or partially cancel the license should business needs change (or if it doesn’t work).

In addition, traditional legacy architectures are often unwieldy, and it can be difficult for businesses to adapt to evolving big data projects or environments.

By contrast, a flexible, licence-based open source environment offers multiple benefits for businesses that want explore big data. Subscription models mean the ability to dip a toe in. And if the licensing is less, as it often is, there is the ability to try without major overhead.

There’s more to it than the cost argument.  Th collaborative, partnership approach to product development associated with open source means the ability to tap into the work of communities of people, potentially accelerating the pace of innovation.

If you think about the latest high-impact big data Apache projects, for example, there are multiple organisations and individuals focused on the development of each one as well as the creation of new projects.

Such are the benefits it delivers that open source is becoming a standard approach in the big data arena. It is helping to drive innovative new technologies like Apache Spark and subsequently Spark Streaming, as well as helping to fuel emerging projects like Apache Beam.

While it is easy for open source vendors to support such projects, it often takes a major ‘crowbarring effort’ for legacy vendors to do so. And, often by the time they do, the rest of the world has moved on.

Just as the cloud has moved from disruptive force into the mainstream, the same process is now happening to open source. That’s why growing numbers of businesses in the big data integration field are adopting an open source first approach.

ABOUT STEVE SINGER//

Steve Singer is ANZ Country Manager, Talend.

Software offered as a subscription is becoming the new standard. In fact, as early as 2015, Gartner estimated that by 2020, 80 percent of vendors would adopt a subscription model.

This change in the way companies use software reflects user demand for flexibility; companies are looking to prioritise variability in their spending based on usage and to ensure they benefit from the value of the software before making a long-term commitment, it is affecting more than business models – it is also driving a faster pace of technological development.

Bringing innovation closer to the market
Forget for a moment how software is billed and consider the value it provides. This is where the real challenge lies: The ability to provide frequent releases that encapsulate current technology innovation and customer demands, while also being easy to manage and scale.

For business users, cloud solutions provide quick access to required resources to get the job done.

By comparison, perpetual license models also allow for periodic software updates – but the rhythm of these updates and the frequency at which they are available to users cannot be compared with the ongoing agility and innovation offered by providers of subscription services.

This is not related to how their software is marketed, but rather to the vendor’s ability to establish a continuous cycle of innovation for its products, the rewards of which transfer to their customers.

Big data and cloud: continuous innovation is the model
The continued growth in the use of big data and cloud technologies is in and of itself, a compelling proposition for continuous innovation. The speed at which these technologies advance requires users to adapt at an unprecedented rate – some have become obsolete in as little as 12-18 months.

It is essential for integration, processing and operating software vendors responsible for these massive volumes of data to get as close to the market as possible, which means complying with key standards such as Hadoop, Spakr and Apache BEAM and aligning with the open source communities defining them.

In practical terms, in order to be successful in a highly competitive, data-driven economy a company needs to anticipate technology changes and create a product roadmap that aligns and embraces them.

Open source technologies – which are backed by the collaboration of a technically adept developer community and various partners – are particularly well suited to a continuous innovation model.

Previously dominant or legacy software models, marked by “proprietary” software solutions and perpetual licensing, took 18 to 24 months to deliver new features.

If you want your business to keep pace with the advances in machine learning, IoT, real-time data streaming analysis capabilities, depending on a model that consists of delivering new versions every 18 months is simply not viable for businesses.

Supporting the emergence of new data uses
Modern solutions for data integration must be at the frontlines of big data and cloud technology innovation. Not only to address customers various and rapidly evolving challenges – including customer intimacy, business sustainability, agility and economies of scale – but also to encourage the emergence of new data uses like streaming, real-time insights and self-service.

In the past, a technology feature could last for years without risk of becoming obsolete (e.g. SQL). Today, the speed at which data platforms become obsolete is mind blowing – for example, MySQL is taken over by Hadoop, which is supplanted by Apache Spark, and who could say what comes next.

Competition is fierce between companies using digital transformation as a strategic lever for performance and competitiveness. The result: Users of these technologies need the ability to easily adapt from one standard to another practically overnight. That’s why it’s so vital to select vendors which are in line with the times. It’s necessary because your business needs to move at the speed of ‘what’s coming next’.

ABOUT STEVE SINGER//

Steve Singer is ANZ Country Manager, Talend.

It seems there are new announcements almost every day about the advancements of machines and their ability to ‘think’.  At the same time, research from Flamingo, a ‘conversational commerce’ company, has found that 77 per cent of Australian consumers are comfortable to very comfortable with the concept of using AI chatbots when interacting with organisations.

We have seen a machine master the complex game of Go, previously thought to be the most difficult challenge of artificial processing. We have witnessed vehicles operating autonomously, including a caravan of trucks crossing Europe with only a single operator to monitor systems. We have seen a proliferation of robotic counterparts and automated means for accomplishing a variety of tasks and all of this has given rise to a flurry of people claiming that the AI revolution is upon us.

However, while there is no doubt that there have been significant advancements in the field of AI, what we have seen is only a start on the path to what could be considered full AI.

Understanding the growth of AI capability is crucial for understanding the advances we have seen. Full AI, that is to say complete, autonomous sentience, involves the ability for a machine to mimic a human to the point that it would be indistinguishable from them (the so-called Turing test). This type of true AI is still a long way from reality; it requires immense computing power, the ability to teach a machine to interpret complex things like emotional responses, and how to generate both intuitive and emotional responses to situations. However, there will be many more practical applications of basic AI in the near term that hold the potential to greatly enhance our lives.

With basic AI, the processing system learns and interprets responses based on “experience.” That experience comes in the form of training through using data sets that simulate the situations we want the system to learn from. This is the confluence of Machine Learning (ML) and AI. The capability to teach machines to interpret data is the key underpinning technology that will enable more complex forms of AI that can be autonomous in their responses to input. It is this type of AI that is getting the most attention. In the next 10 years, the use of this type of ML-based AI will likely fall into two categories:

1. Improvement and automation of daily life: Managing household tasks, self-driving cars and trucks and the general automation of tasks that robots can perform significantly faster and more reliably than humans
2. Exploration and development of new trends and insights: Artificial intelligence can help accelerate the rote discovery and science happening worldwide every day. The use of AI to automate science and technology will drive our ability to discover new cures, technologies, tools, cells, planets, etc., ultimately pushing artificial intelligence itself to new heights.

There is no doubt about the commercial prospects for autonomous robotic systems in the commercial market for aspects such as online sales conversion, customer satisfaction and operational efficiency.   We see this application already being advanced to the point that it will become commercially viable; the first step to becoming practical and widespread. Simply put, if revenue can be made from it, it will become self-sustaining and thus continue to grow. For instance, the iRobot Roomba vacuum cleaner has succeeded as a solidly commercial application of autonomous technology. Autonomous vehicle technology is one of the most publicised and one of the most needed applications of AI. There are an estimated 4.4 million injured or killed in traffic accidents per year in the United States alone; autonomous vehicle technology could almost completely eliminate this and greatly improve availability and efficiency of transportation for everyone.

In addition to the automation of transportation and logistics, a wide variety of additional technologies that utilise autonomous processing techniques are being built. Currently, the artificial assistant or “chatbot” concept is one of the most popular. By creating the illusion of a fully sentient remote participant, it makes interaction with technology more approachable. There have been obvious failings of this technology (the unfiltered Microsoft chatbot, “Tay,” as a prime example), but the application of properly developed and managed artificial systems for interaction is an important step along the route to full AI. This is also am important application of AI as it will bring technology to those who previously could not engage with technology completely for any number of physical or mental reasons. By making technology simpler and more human to interact with, you remove some of the barriers to its use that cause difficulty for people with various impairments.

The use of AI for development and discovery is just now beginning to gain traction, but over the next decade this will become an area of significant investment and development. There are so many repetitive tasks involved in any scientific or research project that using robotic intelligence engines to manage and perfect the more complex and repetitive tasks would greatly increase the speed at which new breakthroughs could be uncovered.

There is also the tantalising possibility that as we increase the capability of our AI systems, they could actually perform research and discover new avenues to explore theories. While this is still a long way away, it could greatly accelerate the discoveries needed for many advancements that could improve and extend our lives.

Anthony Nantes, CEO of ASX-Listed fintech company, DirectMoney, has said, “A seismic shift in consumer expectations is happening globally, affecting every part of the finance industry. The winners over the coming years will be those companies that truly know how to effectively utilise ML and can meaningfully adapt AI to provide better outcomes for their customers. Chatbots are an effective first step in the direction of a brave new world of online relationships with customers, one that will dramatically alter the finance landscape in the years to come.”

As a result, it would be wise to embrace the possibilities that AI offers.

ABOUT STEVE SINGER//

Steve Singer is ANZ Lead at Talend.