How do you eat an elephant? One small bite at a time. This encapsulates why an agile, iterative approach to data warehousing substantially improves the opportunity for success because it enables rapid delivery of early value. It equips development teams to understand business problems directly from those experiencing them. And it provides the headroom to fail fast and learn from easily-rectifiable mistakes.

Back in the gnarly old days, data warehousing projects earned themselves an unsavoury reputation. There are a few reasons for that, among them the simple novelty of an idea which needed time to mature. Another is the typical approach: a ‘big bang’, waterfall-style, all-in style, but one which also had a certain outcome but no real idea of how to get there. It was an attempt to boil the ocean. Or swallow the elephant whole.

These days, things have changed (well, some things have – some of our clients must still use the term ‘data warehouse’ sparingly for fear of opening old wounds). Data is widely recognised for its value. Data warehouses are accepted as an essential foundation to enable analytics and information to flow from data. And the way data warehouses are done has changed. Let’s see how and why.

DevOps before it was DevOps
One of the driving forces behind the development of the data warehouse automation tool WhereScape RED was recognition of a simple principle: you don’t know what you don’t know. The founders realised that the only way to be successful with a data warehouse is to accept that it will be a voyage of discovery. Like any voyage of discovery, there isn’t a roadmap, much less Google Maps, showing the way.

When exploring, taking steps and making missteps is part of the process. If the delivery cycle is short, with rapid iterations and the establishment of ‘minimal viable product’, you can pivot. As new characteristics are discovered in the data, the trajectory of the project can change. Getting to the eventual outcome isn’t a predetermined, singular path (which, in those gnarly old days, might have been pursued with wishful thinking, blind ambition, blissful ignorance or a combination of any one of the three).

“When exploring, taking steps and making missteps is part of the process.”

Instead, the route to success is guided by ‘actuals’ as they emerge. What is the nature of the data? What does it include? How can it be used? Where can it be used? What happens when it is combined? Data is like cooking the elephant; when you mix multiple ingredients, unexpected flavours can result. Mix various data sources and you can discover the unexpected.

While this may sound terribly theoretical, and even risky – after all, show us the executive who is willing to commit funds to a project with an uncertain path to value – this is how the best data warehousing successes come about. Take East Health as an example: the company’s top brass recognised that it had a data problem and a data opportunity. It set off on a path which would take years, but which has delivered unequivocal value. Senior business analyst Jody Janssen says ‘We’ve realised we had no idea of what was in the systems simply because we had no way of getting it out.’

Only once the project was well underway, using iterative processes, was it possible to fully comprehend the data. And fully comprehend what could be done with it.

These days, the process of iterative development which allows for making mistakes and rapidly correcting them is known as DevOps. Back when WhereScape RED was first conceived, it didn’t even have a name – but today, the process of iterative development for data analytics is known as DataOps.

Lessons from The Lean Startup
Back in 2011, Eric Ries published The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Quite a title, but the book resonates because it explains how getting value in front of customers as fast as possible is the best way to get them on board. It’s the same with a data warehousing project, particularly in the context of that hangover from failed initiatives in days gone by. When value is rapidly demonstrated, the business gets keen for more.

Instead of focusing excessively on delivering perfection, it shifts to showing what’s possible. That gets people fired up. It provides the best chance of making the greatest number of people happy. It accelerates engagement between developers and the business community (rather than a standoff which can happen in a data warehouse development team seen as a gated community in waterfall projects).

Because the outcome isn’t set in stone, the very people who will use the data warehouse can influence its direction and functions. All along the way, refinements are made, iterative steps are taken (and sometimes walked back if it doesn’t work – at the cost of a few days, rather than months and millions).

The other lesson is to focus on a pressing, important business problem. Do it at the outset. Solve that one pressing issue and you have delivered value. And don’t be overly ambitious: there’s no point targeting machine learning before the infrastructure is in place. Failing fast in an iterative step is one thing, failing on a major objective another.

Software and people
Tackling a data warehouse project iteratively depends to a large extent on having the right tool. Software like WhereScape RED encourages the method because it has the frameworks and automation built in; there is a certain amount of data plumbing which must be done to enable data to flow. RED works on the Pareto principle: it does 80 percent of the work, so the developer can focus on the 20 percent where the real difference is made (this is how RED makes developers 4 to 10 times more productive).

Approached the right way, data warehousing projects are transformative. They are how the new oil is extracted, refined and turned into fuel for your business.

ABOUT MARTIN NORGROVE//

Martin Norgrove is CTO at NOW Consulting, a data and analytics services company. With a BSc in Chemistry and Physics from The University of Auckland, Martin initially started working life as a lab technician at Carter Holt Harvey before discovering his passion for all things data. He’s worked for numerous high-profile brands including Spark, Z Energy, ASB, Auckland Council and Lotto, and thinks the future is very bright.

A question mark has also surrounded security: for enterprises which are using data warehouses, the integrity of information is paramount. But with these issues addressed, the time is ripe for your organisation to, at the very least, test the waters with cloud data warehousing. Why? Because the benefits and advantages are so clear that if you’re not doing it, you are missing out.

Maturity is often just a question of how long something has been in the market. That ‘time in the saddle’ gives vendors the room to experiment, identify and iron out issues, and shape up value propositions. It also gives early adopters the opportunity to get to grips with a new way of doing things.

With data warehousing in the cloud, the major steps to maturity have come in the past 12 to 18 months. The big vendors in this space, including Microsoft Azure with SQL Data Warehouse and Amazon’s AWS Redshift, are highly viable solutions.

More than that, they are now ‘battle proven’; particularly in the last six months, we’ve seen several serious enterprise workloads successfully move into the cloud.

In simple terms, that means when you go ahead and try a data warehouse in the cloud, you will not be a ‘test case’ as there are thousands of other organisations which have done it before and which are seeing the advantages. The big question of ‘security’ is resolved – not just for data warehousing, but for cloud computing generally. The companies which offer cloud solutions, after all, invest an exponentially larger sum into security than any of their customers can. Why? Because their very business depends upon it. If Xero and other leading organisations have done it, so can you. “It is not an overstatement to say the cloud has been crucial to Xero’s success,” Andrew Jessett, Xero’s GM of IT has said. “We could not have succeeded without it.”

In other words, trust in the platforms is established, demonstrated and proven.

Substantial benefits
Let’s look at the advantages of a cloud data warehouse. Most of the pluses are somewhat generic to any cloud service, but it bears repeating in the context of the data warehouse: cost, efficiency, flexibility, speed (or time-to-value) and simplification.

Starting with cost and efficiency, enterprises with a certain set of problems recognise that cost and efficiency are major challenges on the path to value. When the data warehouse is in the cloud, it can be ‘stood up’ in a matter of minutes with a few mouse clicks. There is no need to procure, install and then maintain enterprise hardware (an exercise which typically requires approval of capital budget, delivery times and installation, all of which can take months). The software, be it Azure SQL, Redshift or anything else, is packaged with the infrastructure, all in one go. The saving in hassle, time and money is immediate and extreme.

Then there is the question of software license costs. In the cloud, that drops by four to five times when moving from, say, a traditional data warehouse to Redshift. It isn’t just the immediate cost benefit either: traditional data warehouses can be expensive and slow to upgrade and update. It can even be difficult to unlock new features.

What of performance, or scale? Traditionally, you’d need to kick off that tin procurement exercise if you needed more of either. With the cloud, more capacity and performance is on tap – it scales linearly. We’ve seen a process which takes 80 minutes, drop down to 10 minutes. For businesses processing data multiple times per day, the advantage is obvious.

Costs aren’t just a factor of establishing the infrastructure; the skills necessary to get to work on populating the data warehouse must be factored in, too. As far as Microsoft is concerned, the skills are entirely translatable: SQL is SQL. There is a common developer experience, and when a tool like WhereScape RED is introduced into the mix, data translation and automation makes for a highly efficient (and fast) data warehouse setup (and for all the WhereScape RED fans out there, the experience in RED is almost identical).

It’s all about the analysis
By now, the time-to-value advantage should be making itself obvious. The point of a data warehouse isn’t to have a data warehouse but to have it as a foundation on which to conduct analysis and get outputs of actionable information and insights. When the data warehouse is stood up in the cloud rapidly and at a dramatically reduced overhead, that means the good stuff starts happening far faster. Business analysts can do their thing with the cloud data warehouse in days, rather than standing by and hoping something comes out of it in weeks, months or perhaps, as is sometimes the case, never.

That’s particularly relevant right now, with all the media attention on machine learning, artificial intelligence and other advanced analysis techniques. These may be in the ‘marketing exercise’ phase of maturity, but just like cloud data warehouses took a little time (not a lot, mind) to mature, so too will these concepts shortly be ready for prime time. Putting them to work and getting valuable outcomes is enormously more viable when your data warehouse is in the cloud; already, some algorithms can simply be plugged into your data warehouse. Expect a lot – like really, a LOT – more as these concepts shoot up the maturity curve. With the basics of a data warehouse in the cloud, you are equipped to do the fancy stuff fast, inexpensively and – importantly – experimentally. With such a low cost, you can go right ahead.

Enabling innovation
Another of the popular words in the press these days is innovation. There’s no better way to take the excitement out of a potentially new way of doing things than telling the eager data scientist he’ll need to wait a few months before the means to test his latest hypothesis is established. The cloud delivers immediacy. Had a great idea which needs data analysis to test it? Stand up a data lake right away. Test, experiment, verify, get into production faster, at lower cost and all the while keeping smart staff members enthused. If it works, fantastic. If it doesn’t, no worry – failing fast allows for iterative improvement without breaking the bank.

That helps foster a culture of innovation, where people in your teams are equipped to try new things.  If it doesn’t work, close it down – and you aren’t stuck with a bunch of servers and more shelfware gathering dust.

What’s not to like
With low barriers to entry, proven use cases and the ability to do much more with data, much faster, the case for data warehousing in the cloud is a strong one. The bottom line is a simple one: if you aren’t at least testing data warehousing in the cloud, you are losing out.

ABOUT MARTIN NORGROVE//

Martin Norgrove is CTO at NOW Consulting, a data and analytics services company. With a BSc in Chemistry and Physics from The University of Auckland, Martin initially started working life as a lab technician at Carter Holt Harvey before discovering his passion for all things data. He’s worked for numerous high-profile brands including Spark, Z Energy, ASB, Auckland Council and Lotto, and thinks the future is very bright.