Common reasons why so many data projects fail to deliver

Where you’re going wrong and how to pivot towards data engineering success

Back in 2017, Gartner estimated that 85% of big data analytics projects fail. Five years later, that needle has only moved ever so slightly, with 20% of analytic insights projected to deliver business outcomes in 2022.

Those are some fairly sobering statistics, considering that big data and business data analytics markets in Australia and New Zealand have more than doubled in less than five years – and things aren’t slowing down.

If the majority of data analytics projects fail, where are they going wrong and, more importantly, how do you avoid making the same blunders?

Unclear project objectives

This common mistake sits at the top of the list for a reason. Without clearly defined data project objectives, it’s near impossible to achieve project success – it’s like trying to build kit-set furniture in the dark! The same goes for data projects with unrealistic goals or a lack of meaningful value to the business.

The solution
To mitigate this risk and start your data project on the right foot, you must spend enough time upfront outlining the aim, objectives and problem statement for any analytical work, so that all major stakeholders in the business are on the same page. Doing a deep dive into truly understanding the underlying problem will help define the ‘North Star’ of your data project.

Jumping in without the right data

Without the right data, your data project is doomed to fail. Whether it be the wrong data, ‘dirty’ data or a lack of data governance, the result will be the same – wasted time and resource, and the risk of making business-critical decisions without the best possible information.

The solution

With the ever-increasing level of data that organisations collect, it can be a challenge to not only access the data you need but also overcome a mountain of issues. For example, is the data clean, secure and legally usable? That’s why robust data governance is essential, alongside data monitoring tools, to maintain the health of data moving forward so your data projects ultimately deliver actionable insights.

Failed implementation

Failing to gain traction or buy-in from the business before a data analytics project can move forward isn’t ideal, but it’s not the worst-case scenario. What’s worse is when a business spends time and money on a project – and ends up not implementing it or seeing value in the results. This leads to unhappy stakeholders, tighter budgets and further apprehension about using analytics.

The solution

Avoiding this comes down to the planning stage. As mentioned, clearly defining your project objectives – what success looks like and why the business should care – will lead you down the right path. Taking an agile approach will ensure you stay on that path. Breaking large data projects into sprints lets the business see constant returns and viewing the entire project in stages with a priority focus will help maintain momentum and keep stakeholders happy.

Technology is valued more than the outcome

As data people, we sit firmly in the ‘tech is cool’ boat, but it’s easy to get swept up by new and exciting technology. It’s also common for businesses to over-engineer solutions to problems that would get more benefit from a simpler solution. In data project land, that can often look like experimenting with new technologies or methods that don’t solve the problem, adding even more intricacies to what’s probably an already complex feat.

The solution

Keep exploring new technology solutions, but make sure that whatever you integrate fits the outcome that you’re trying to achieve. It’s not about needing to have technology, but rather, will this technology help you solve a problem? Furthermore, any new technology must not only be useful but also flexible, so it can add more value in the future.

Relying on one internal set of eyes

Gone are the days of the lone-wolf data scientist executing a complete data analytics project alone. However, while data projects require the right mix of talent across teams within an organisation, businesses all too often hand over projects to one internal person or team, which can be risky for several reasons – they might leave the company, have conflicting priorities (prolonging timeframes for delivery) or unintentionally do things incorrectly.

The solution

Subject-matter experts are great, but they need the support of other internal teams and even third-party experts. Make sure teams are equipped with technology as well as the knowledge and skills to leverage said technology. Keep communication lines open with business stakeholders and focus specifically on data ROI measures to ensure they remain engaged.

Relying on an external partner

Depending on a third-party expert alone isn’t the right answer for avoiding failure either. Many organisations have large and capable internal teams that can handle the day-to-day analytics work, but the right partner can add value by supporting the business to gain momentum on new, difficult and complex projects.

The solution

The sweet spot for true business acceleration is a combination of external expertise and building internal capabilities. Start with small, nimble internal teams, bolstered by external partners where talent gaps exist and point them at tangible use cases that affect numerous internal stakeholders, so the power of analytics can flow through the entire organisation.

Don’t be afraid of project failure – tackle it head-on

With a wealth of documented failures, it can be hard to approach any new data project without feeling like you’re on the back foot. But just as there are projects that fail, there are projects that win, and they all share a common thread. Their objectives are clearly defined, they work with clean, useable data, they use technology solutions that add benefit and they empower internal teams with the support of a great data analytics partner.