Published on the 23/09/2021 | Written by Scott Castle
Moving beyond traditional BI calls for a modern data team…
The global pandemic has driven home the fact that data is vital to the success of every organisation. Companies across Australia and New Zealand (ANZ) are realising the importance of scaling and growing their analytics capabilities, something that has become even more critical in the ‘covid’ era.
In fact, 67 percent of businesses across A/NZ now view BI and analytics programs as more or much more important to business operations now than before the pandemic.
But what does it take as a data engineer to keep up with the innovation needs of today’s fast-paced businesses? And are we expecting too much from today’s analytics talent?
‘The solvers of all business problems’
Despite widespread recognition of the value of data in business, budgetary constraints, skills challenges, education around data, and the best use of employees’ time, remain key challenges. There is also the risk of data scientists being seen as ‘the solvers of all business problems’ – and they become overloaded with irrelevant questions.
“A number of facets are key to successful data and analytics, and they’re all very much related to people.”
In fact, what we’re seeing more than ever is the data analyst, and the data analytics teams, suddenly becoming the ‘perceived source of all business decisions’, while the CEO, the CTO, the CFO are just ‘hammering them’ with questions.
Yes, there are certainly a few questions that are really important and require a good analyst or data scientist to help solve. But there’s also a bunch of questions that can be solved by the questioner going back into the BI tool and pulling up their own chart.
In reality, this skill a data engineer has is a limited resource, so companies have to start thinking about how to prioritise and send the right questions, the most impactful and the hardest question, to the data team.
It’s important to match the right contributor to the right type of question and not just ‘throw’ data scientists into a lot of basic reporting. Companies should also not use data analysts to try to solve massive predictive problems or use either one of them to solve low-level tasks like routine group and count.
In this regard, understanding the role of data for business success is key while making the most of the talent and skills a data analyst has within a business. The right BI and analytics tool can make a good difference, however business must also consider adequate resourcing and skills to best utilise these tools.
Avoid the pressure to ‘attack data with one tool’
There’s pressure on data teams to try and get all data into the same format, and to ‘attack it with one tool’.
There are two challenges in this. Firstly, the technical challenge – not every question is answered with one particular tool, there’s no one analytics that will always work. Businesses need a whole tool chain that changes based on the question you’re asking.
Secondly, there’s the over standardisation of the data culture and the people. When you teach a single methodology for analysing data, for approaching questions and getting access to the answers, and what kind of answers you’re looking for, you end up with single results.
The more you over-standardise, the more you force every answer to look the same. It’s when you have unusual approaches, or two or three approaches to the same question, which allows you to really have confidence in the data and you don’t get that with over standardisation.
Interestingly, Dr Alex Antic, voted as one of the top 10 analytics leaders in Australia by IAPA, also aptly points out, in order to not over-standardise it’s important to have a suite of tools, an open-source framework, as well as any vendor-specific tools,
“This also gives analysts scope to be able to use the tools they prefer, depending on the particular problem at hand,” he says.
Critical skills of a modern data engineer
According to Dr Antic, there are a number of key elements that support great data talent within organisations:
“There are a number of facets which are key to successful data and analytics, and they’re all very much related to people,” he explains. “Having the right people in the organisation in the right roles and frequencies to be analysts, data scientists, and software engineers to enable capabilities from a technical viewpoint is vital.”
A strong technical leader, who understands the solutions and can guide the team is also key, he adds. This leader needs to be adept at finding the right problem to solve and able to work closely with people as an evangelist and trusted advisor within the organisation.
Understanding the power of democratisation of data within the organisation is also vital, Dr Antic adds. This process supports education (such as improving data literacy) and helps create and support the data strategy for the organisation so that the C-suite has a clear understanding of how to use data.
“This all contributes to a culture of innovation, experimentation, and exploration,” he explains. “It’s vital to collaborate, share data, and knowledge throughout the organisation.”
Having a solid grasp of ethical use of data/privacy is also a must in today’s compliance-driven and highly regulatory environment.
“It’s becoming much more important for organisations to be able to understand ‘responsible use of AI’, how to deal with bias, and the ethical concerns around fairness in data, as well as how it will be used by the end-user,” Dr Antic adds.
Finally, a great data engineer needs to have great communication skills, as it’s communication that brings all these elements together.
“Communicating the benefits of a strong analytics program empowers you to build up a winning coalition, so people, analysts, and business leaders can understand and embrace the benefits of analytics internally,” he says.
The evolution of the modern data team
The future looks exciting. Companies that are moving beyond traditional BI are those that are disrupting markets. They’ve invested in modern data teams to optimise insights across the organisation, driving growth and ROI in the process.
Where data teams of the past may have had three traditional participants: database administrators responsible for data warehousing capabilities and capacity, data analysts focused on data modeling, and BI architects in charge of building dashboards and related self-service reporting capabilities, today the team is so much more.
Typically a team that partners with the business but has its own specialised skills, this group does everything from outlining business rules to modeling data to building single sources of truth.
Most importantly, these individuals provide ad hoc analysis, the kind of assessment that you don’t really get from traditional BI. Ask business questions like “what should my revenue model be?” and the group will test its hypotheses to surface unexpected insights. They may apply machine learning or some statistical and predictive technologies to determine patterns or gaps where your organisation simply does not have ideal data.
On top of this, can your data analysts blend model data with raw data from multiple sources? This is a nuanced but worthwhile question. Ideally, your data team can take raw data coming in from a new system and join it with existing model data, such as account, opportunity, or service data, to answer questions that may not have been anticipated in a current data model.
If data is blended from multiple sources into a single warehouse, does that process occur before or after it arrives at the warehouse? For example, a more modern approach would blend the data post warehousing.
The key takeaway is that, unlike a traditional BI team that provides only a data model, today’s data team operates as a core function of the business and recommends a strategy built on data. The group has a seat at the table, offering informed opinions – based on data – about what the company should be doing.
ABOUT SCOTT CASTLE//
Scott is the VP & GM for Cloud Data Teams at Sisense. He brings over 25 years of experience in software development and product management at leading technology companies including Adobe, Electric Cloud, and FileNet. Scott is a prolific writer and speaker on all things data, appearing at events like the Gartner Enterprise Data Conference, Data Champions, and Strata Data NYC.
