In business, we primarily work on regular, repetitive tasks. Projects that fall outside business as usual are usually both more complex and require more expertise than we have inside the team.
When faced with this situation, the key question is, “Should we make it, or should we buy it?”
Was it worthwhile investing in building up the skill, capability, and expertise internally, or would we be better off hiring external resources? Hiring experts from outside the business is typically the best solution for risk mitigation for a project like a data migration, which does not repeat frequently.
You work with data; you know data migration is complex and contains multiple business risks. Principle among these is maintaining data integrity and accuracy. The hoped-for outcomes from a data migration include efficiency, better data security, and enhanced accuracy.
Undergoing data migration is no easy feat, especially if you’re a large organisation with multiple data sources and complex systems that you want to integrate into one single source of truth. You will probably find that once a data migration is proposed, many additional use cases get added to the scope of work—what may have started as a plan to merge data from multiple systems morphs into a systems upgrade, cloud migration, changing data centre or changing database vendor.
A lot also depends on the type of migration you choose to do. Which platform you migrate to, your data quality, and what you intend to do with it all influence how the process looks.
We’ll explore the challenges, benefits, and our leading tips for undergoing a data migration in your business. While every data migration has unique quirks, this article should help you better understand what to expect.
You might consider a data migration if:
Data migration is a long-term IT strategy that makes future analytics and campaign work much easier. Done well, it improves orchestration, customer experience, and operational efficiency across your organisation.
The first thing you should know about doing a data migration is that you should never tackle it in its entirety. There are too many unknowns that could trip you up or change the scope, and the task of trying to address them all at the beginning would be gargantuan.
When planning a data migration, the first job is to build a team of experienced individuals who together contribute everything you anticipate needing for the whole project. Every data migration plan has three phases – data assessment (what you’ve got, dependencies and constraints, data audit and security concerns), project scoping (detailed requirements specifications, resources and whether to trickle-migrate or big bang), and data backup (your failsafe in case losses or corruption happens during the migration).
Regardless of your end goal, we always recommend that you break the project down into phases. This enables you to build more robust value hypotheses and create learnings around the process, systems, and project team that you can apply to the wider migration. It also means you can fail fast – any mistakes you make will be less likely to affect the project.
To succeed in this proof-of-concept stage, you must choose a suitable data sample to migrate. A sample that is too small could mean that you ensure quick success that does not reflect what it will look like to migrate the wider system. One that is too large also defeats the purpose, as you would essentially experience the same pain points you would face doing it all at once.
For example, an end-to-end migration of 10% of your data can add value to your business and allow for a better cost estimate for the wider migration. Spending $60K on your proof-of-concept can tell you that the broader migration will likely cost $500 - $550K. Without a proof-of-concept, your initial cost estimate would be more expansive—say around $400 to $800K—and, therefore, financially much riskier.
Early on, when approaching your data migration, you will need to gain a thorough, granular understanding of what your current system is doing. You also need to have a very clear goal of what you need the new system to do. The selected target system also needs to be assessed and compared. Both require detailed documentation to deliver the best possible scope of work.
This process presents a perfect opportunity to review what you do and what you don’t need to migrate. You might be surprised by the number of tables, systems, and processes that have become redundant over time but still take up a lot of resources.
Armed with this knowledge, you can start planning the migration itself. Once you have completed your proof-of-concept, you can then apply the learnings to a more concise and accurate plan for the next phase. Whether that’s another component like your proof-of-concept or the remaining system will be determined by the scale and complexity of your project.
Data migration is a big job that involves many parties. One of the most important roles in ensuring your migration gets across the line is the project manager. You will need someone who has technical knowledge and an understanding of the interdependencies to ensure all the stakeholders are informed – especially the budget holder.
If you’ve never done a migration before or tried and failed in the past, your best bet is probably to work with an agile, experienced partner who can anticipate problems before they occur. Whereas your internal analytics team manages business as usual with skill, as we said in the opening sentence, a data migration falls outside their expertise.
Due to the scale of this kind of project, it’s impossible to fully specify and scope out a migration in advance – which can be frustrating. That’s why we always recommend breaking down your project into phases and staying flexible. Datamine has led dozens of client data migrations, and we have never seen a result that looks the same as the specifications.
It’s also critical to take the time to fully understand the pros, cons, and capabilities of the CDP systems you want to migrate to. A misunderstanding of what your platform can do will eventually mean you must shell out more time and money to develop the other areas. If something you require is missing from the platform, budget the costs of that extra customisation.
We get it – a data migration is daunting. With the right team on the job, a careful project plan and risk mitigation, your future data “nirvana” could be ready for use later this year.