The data secrets behind Xero’s cloud migration

Published on the 28/11/2018 | Written by NOW Consulting

Video_Nathan Griffiths_Xero_NOW Consulting
icon_case study

AT A GLANCE

Xero

INDUSTRY

  • Business Software

BUSINESS OBJECTIVE

  • Improve team effectiveness
  • Accelerate solution delivery
  • Inform better business decision making
  • Enable scalability

SOLUTION

  • Migrate the Xero data warehouse from large on-premise SQL Server to cloud-based AWS Redshift using WhereScape RED data warehouse automation and the services of NOW Consulting.

BUSINESS BENEFITS

  • Project accelerated using WhereScape RED, reducing time to value for Xero
  • Unlimited on-demand scalability
  • Significant savings on data warehouse infrastructure costs
  • Improved solution delivery for data team

FOR MORE INFORMATION

NOW Consulting
W: https://nowconsulting.co.nz/
E: contact@nowconsulting.co.nz
T: +64 9 358 8521 or +64 27 433 6463

Moving 59 billion records to AWS Redshift no job for a forklift...

One of New Zealand’s biggest international technology success stories, and an organisation which breathes data, leveraged NOW Consulting and the WhereScape RED data warehousing automation tool to move its data warehouse onto the Amazon Web Services cloud. The first-of-its-kind migration meant the cloud accounting provider added its BI stack to its new Platform as a Service provider with minimal impact, while preserving the logic and functionality of an established on-premise data services platform.

“If we didn’t have RED, we simply would not have the tools to go to work on our data.”

Needing little introduction, Xero was founded in 2006 and is one of the world’s fastest growing software as a service companies. It leads the New Zealand, Australian, and United Kingdom cloud accounting markets, employs a world-class team of more than 2,000 people and was identified by Forbes as the World’s Most Innovative Growth Company in 2014 and 2015.

With a million customers and growing, data is vital to its continued success.

Situation
Xero has appreciated the value of data since its inception, and its engagement with NOW Consulting goes back to ‘early days’, confirms Nathan Griffiths, Data Services Platform Lead at Xero in Auckland.

“Data is a key business asset. We recognise this and recognise the value in the data we hold in terms of providing services back to customers, creating our Small Business Insights report, and even in pages on our website which provide high-level overviews of business performance in the regions where we operate,” he explains.

“Our data is used by government analysts to assess trends within business, and it is used to inform innovation and provide services within Xero products. Essentially, data is a fundamental component in the services we offer.”

“The BI team where I work has been around for 5 or 6 years, so that’s more or less the time period in which we’ve used WhereScape RED for building the data warehouse.”

When Xero opted to move to Amazon Web Services (AWS), the implications were enormous: with more than 59 billion records, 3,000 apps and 120 databases representing all data and applications for its 862,000 customers globally (at the time), the sheer scale of the migration is clear. “It meant a huge rearchitecting of all infrastructure and systems which involved changing from SQL Server to AWS Redshift as the database platform. It was a major undertaking,” says Griffiths.

The data warehouse was part of this exercise; built over time it represented a considerable investment as it was created and constantly modified to meet changing business information demands.

But Xero had a couple of aces up its sleeve which would ease the transfer of this structure to the new platform.

Solution
The first step was an engagement with NOW Consulting, which performed a proof of concept to show that it was technically feasible to move from one platform to another while preserving the logic, processes and code of the data warehouse. “NOW delivered on that aspect successfully, then provided additional manpower for the actual migration,” says Griffiths.

One of the major challenges faced was what he describes as ‘a hefty SQL Server’ hosting the existing data. “We just could not recreate a system of that size in Amazon; we couldn’t just lift and shift, so we needed a new solution in Redshift which was an unknown.”

“If we didn’t have RED, we simply would not have the tools to go to work on our data.”
Nathan Griffiths, Data Services Platform Lead at Xero

He glosses it over as ‘a fair bit of risk and a lot of work’. “With NOW’s help and with WhereScape RED as the tool to do it, we were able to reduce the complexity and risk to a point where it was feasible.”

There was a little pressure, too. “There wasn’t any question of it not being done as we were closing off and had to have a solution in place; the consequences of no solution would be a lot of internal analytics and reporting would have ceased to work, it would impact our Customer Experience teams, our sales and marketing teams and even our product teams.”

NOW developed the necessary functionality within RED which enabled connecting the existing data warehouse to the AWS Redshift database. “This was a new feature at the time called Templates; instead of code generated in WhereScape itself, the template has variables specifically for Redshift which allowed extremely flexible code generation. This was the key to moving to the new platform,” says Griffiths.

Results
In short, Griffiths credits NOW Consulting and the WhereScape RED product for enabling a successful migration with minimal disruption of a key asset in Xero’s data-driven approach. “It’s been extremely valuable, not only in terms of the migration itself, but after the fact,” he confirms.

One of main differences between SQL Server and AWS Redshift, he explains, is that the former has a scheduling agent. “By contrast, Redshift is just a database with no other features; you have to provide your own. RED takes that over to run code on the database and do certain things, so if we didn’t have RED, we simply would not have the tools to go to work on our data.”

Despite its name, Redshift also does not provide any tools to manage data migration. RED plugged that gap.

The key features, Griffiths says, include RED’s provision of visual data representations and its ability to automatically generate code. “RED is used daily by our BI developers who build data models, data marts and other models. The code is reliable and repeatable. That’s hugely useful, you can easily schedule and manage jobs and dependencies.”

Where in the past there were multiple environments, versions, tools and models, today it is all done within WhereScape RED and on AWS Redshift. “That’s brought structure and reliability to the development process and our data initiatives,” Griffiths notes. “The ability to generate custom code by building and modifying templates has been very successful. It has allowed us to take advantage of specific Redshift features; matching the platform with the development environment and scheduling capabilities of WhereScape has enabled us to make the leap to AWS with minimal pain.”

Source: This article was originally sourced from NOW Consulting

FURTHER READING


About This Vendor


More Case Studies

Abano Health Microsoft Azure video

Drilling for value: Dental group does ‘Datability’

October 25, 2018 | NOW Consulting
Abano Healthcare migrates data warehouse to Azure, achieves cost savings, accelerates innovation and data value...
NOW_Wherescape_RED_East_Health

How data insights contribute to a healthier population

March 7, 2018 | NOW Consulting
East Health pioneers data culture for information-driven healthcare decisions…
Lotto_NZ_analytics

How data analysis underpins Lotto NZ’s operations

July 5, 2017 | NOW Consulting
Visual reporting tools are delivering insights to senior execs and local dairy owners alike…
Auckland City of Sails

City of Insights

June 29, 2017 | NOW Consulting
How Auckland Council leverages data to drive daily business…
Wherescape_University of Auckland

University of Auckland makes intelligent choice on data

May 18, 2017 | NOW Consulting
A new data warehouse has delivered agility, speed and transparency for one of NZ’s oldest institutions…


Follow iStart to keep up to date with the latest news and views...