data migration what to expect

What to expect when doing a data migration

Undergoing a data migration is no easy feat, particularly if you’re a large organisation with multiple data sources and complex systems you want to integrate into one single source of truth. Datamine Partner Matt Wilkins has helped clients through many data migrations over the years, and has the battle scars!

Use are Matt's insights on both the challenges and opportunities presented by data migrations, and his suggestions for organisations looking to do a data migration.

Marketing: So how many data migrations have we done at Datamine?

Matt: I actually counted up recently, and we’ve created 28 marketing CDPs alone - so probably somewhere around 50 data migrations in total? They’re substantial and complex projects, and we’ve definitely felt a lot of pain, both on the client side and our side.

Marketing: What are the biggest pain points, typically?

Matt: I’d say one of the recurring issues we often see is that it’s impossible to fully specify and scope out a migration in advance, which can be frustrating for clients. You need to be super flexible and responsive, because there will always be something that comes up and forces you to pivot. IT's easy to get a false sense of security by over-specifying ahead of time. For context, I’ve never done a migration where the specification perfectly matched the end output, and despite that clear fit, we needed more budget and time than anticipated - in fact, it’s usually safe to double your time allocation from what you think it’ll be. This might seem like an over-exaggeration, but it’s not!

One of the other major pain points we see is actually in choosing the CDP system you’re migrating to. I’d say more than half the time, the organisation has picked a new system without fully understanding its pros, cons and capabilities. This means they enter the migration process thinking they’ll be able to do certain things, but in the end they have to shell out more time and budget to get those other areas developed. It’s important for companies to fully understand the functionality of the system they’re migrating to, and if there’s stuff missing, how much that extra customisation will cost.

Marketing: Sounds like a convoluted space. But would you say the benefits of doing a data migration typically outweigh the challenges?

Matt: I think it’s usually worth the struggles. Yes, there’s a lot of pain throughout the process, and the end result might not be exactly what you had initially thought you were going to get, but it’s a ‘long game’ play that enables future analytics and campaign work to be much, much easier. If all goes well, you get improved orchestration, better customer experience and an increase in operational efficiency for the organisation. We’ve seen clients have huge success here.

Marketing: What kind of team do you need to do a data migration?

Matt: I think one of the most important roles is the project manager who can manage the migration and liaise between all the involved parties. This person needs to have technical knowledge, otherwise you’ll have to put technical capability around them in order to make sure they’re equipped to champion the project. They also need to understand the interdependencies (from a data point of view) in order to ensure all stakeholders are informed. Internal documentation and specifications need to be shared amongst everyone involved, and people can’t be operating on their own. No one should be making decisions in one isolated area. I’d also say that the person making any decisions about the scope and activities must have a strong connection to the budget holder. I’ve seen in way too many projects decisions being made without prior consultation of the budget holder, and that doesn’t usually go well.

If you’ve never done a migration before, or if you’ve tried and failed in the past, your best bet is probably to work with an agile, flexible partner who can anticipate problems before they occur. Where other types of analytics work might be doable internally, we’ve found that internal teams lack the battle scars and experience needed to grapple with what might seem like an easy process but is actually quite convoluted. Regardless of whether they’re internal or external, data migrations absolutely require subject matter experts who are available and actively working on the project.

Marketing: Do you have any other advice for organisations looking to undertake a data migration?

Matt: As I touched on earlier, it’s important to focus on provisioning enough budget and understanding the right level of up-front specification. Whenever we talk to people who are looking to do this, they’re concerned about going over budget or having the project blow out of control, and the best way to avoid this is to allow yourself enough flexibility at the outset.

Beyond that, I’ve got three other pieces of advice:

1. Trying to migrate everything in one massive switchover is really risky and hard, particularly if you’re dealing with operational systems and need to switch from one to the other with no fall-back period. We suggest building the new datasets and QA-ing as you go, then layering additional data over that over time so that the ‘switchover’ is just adding the final bits of data to a system that’s already set up .

2. Make sure if you’re combining multiple systems into one that you understand the quality of each system individually, as well as where the data conflicts. You can get 95% of the way pretty quickly, but this step often takes heaps of time.

3. If you’re migrating your data to a system that has specific requirements and you’ve got other useful data that doesn’t map into those requirements, don’t get rid of it! You should have a plan for keeping it somewhere accessible so you can still do analysis on it in the future.