The situation
The Australian Energy Market Operator’s (AEMO) vision is to deliver energy security for all Australians. It administers and operates the country’s wholesale national energy markets, with around 18 terabyte of data stored across a myriad of different on-premises systems, including some high-volume transactional systems.
With energy consumption on the rise around the world, recent heatwaves saw Australians use more electricity than ever before. This anomaly highlighted how difficult it is to gain insights into how electricity grids cope with the ebb and flow of energy consumption across all consumer and commercial meters.
The opportunity
Historically, AEMO’s data was used in isolation across its many systems. Different business units acted in silos, generating their own reports without a consolidated, single source of truth, meaning energy grids couldn’t be compared and increases or declines were not noticed. An internal initiative to aggregate data in a pseudo data warehouse was carried out, however the volume of data proved too much as certain queries were taking days to execute and ministerials were taking weeks or even months to answer.
Empired helped AEMO use big data to drive smarter ways to save energy consumption and improve forecasting accuracy across Australia. This solution is key to planning how the network will look in the future and the infrastructure required to ensure continued high levels of service to energy consumers.
The technology
Using Microsoft Azure’s best-of-breed data platform with rich ecosystems of data components, catering for AEMO’s evolving needs, Empired assisted AEMO in tapping into their energy consumption data. This data intelligence solution allows data to be analysed right across the country and across grids every two minutes in a flexible way.
Energy data, population data and weather data can all be factored in and assessed to forecast a future approximate for the next 20 years in 30 minute intervals. Furthermore, Azure provides the elasticity to be scalable depending on AEMO’s needs, achieving cost savings by scaling up only when required, then pausing or powering down when not in use, such as weekends. This is one of the most complicated and sophisticated uses of data intelligence in the world.
Empired designed and built big data warehouses using Azure SQL Data Warehouse, Azure Data Lake, Power BI, machine learning, Python, Spark, R and Spark-R, Data Factory and HDInsight. The data flows were designed to load data from AEMO’s on-premises systems to the Azure-based data warehouses, automating the build of the data flows from more than 700 tables (some larger than one terabyte) from onpremises systems in Azure. Australian-based data centres ensured all information was stored in-country.
“It is magic. But it is a true story and an enormous success story.”
Ben Johnson, National Business Manager, Data Insights & Integration, Empired
Source: This article was originally sourced from Empired
