What 17 Data Centers Taught Me About Disaster Recovery

Disaster recovery planning is where organizations tell themselves comfortable lies. I spent nearly a decade managing global data center operations at Accenture — 17 sites, 800,000+ users, a budget exceeding $10M — and I’ve seen how that gap between documented DR plans and actual recovery capability plays out under pressure.

Here’s what I learned.

The RTO You Document Is Not the RTO You’ll Hit

Every DR plan has a Recovery Time Objective. Most organizations test that RTO once a year, in a controlled exercise, with the best people available, fully rested. That’s not a disaster — that’s a rehearsal.

Real disasters happen at 2 AM on a Friday before a long weekend. The engineer who knows that system best is on vacation. The runbook was last updated 18 months ago and references three servers that were decommissioned. The failover scripts work — but they assume the secondary site has current data, and it turns out replication has been silently broken for six weeks.

The lesson: Test your RTO under adverse conditions. Run exercises with the B-team. Inject failures into your exercises. Your documented RTO is a marketing number until you’ve hit it in a surprise test.

Replication Is Not Backup

These are not the same thing and conflating them is how you lose data. Replication gives you a current copy — but if corruption or ransomware hits the primary, replication propagates it to the secondary before you can stop it. Backup gives you a point-in-time recovery option.

At scale, you need both: synchronous or near-synchronous replication for RTO, and air-gapped backups with retention periods for RPO in the corruption scenario. We used Zerto for replication across our data centers — journal-based, continuous, with granular recovery points. But Zerto didn’t replace our backup platform. It complemented it.

Know what each tool protects you from. Replication protects against site failure. Backup protects against data corruption. Neither protects you against an insider threat that operates slowly enough to poison both.

Dependencies Are the Real Failure Mode

The server didn’t fail. The application failed because the database it depended on was on a VLAN that wasn’t included in the failover scope. The database was on the failover list, but it couldn’t connect to the authentication server because that was in a different tier with a longer RTO.

In complex environments, DR failures are almost never about the primary component. They’re about the dependencies you forgot to map. Before I managed a DR exercise, I required the team to produce a dependency graph — not just the application list. Every external integration, every authentication dependency, every DNS resolution path.

It takes time. It surfaces uncomfortable conversations about systems that weren’t supposed to still exist. Do it anyway.

The 17 to 10 Consolidation

During my time at Accenture, I led a data center consolidation from 17 sites to 10. Every consolidation is also a DR exercise — you’re migrating workloads across sites and the business keeps running through it. What I learned: the technical migration is the easy part. The hard part is the organizational dependencies you don’t control.

A team in Germany has an application that calls a legacy API hosted in a Frankfurt data center you’re decommissioning. They didn’t know the API lived in Frankfurt — it was abstracted away. Now you’re having a conversation about migrating or replacing an undocumented dependency with three weeks left in the project.

Map everything before you move anything. Assume the map is incomplete when you start. Build in discovery time.

What Actually Matters in DR

After managing these environments for years, here’s my short list:

• ▸Runbooks that a stranger can execute at 3 AM. Not the expert. The person on call who’s never touched this system.
• ▸Tested communication trees. Who calls who when something breaks? Is the contact list current? Does it include vendor escalation paths?
• ▸Regular surprise tests. The only way to find gaps is to look for them before an incident does.
• ▸Honest RTO/RPO conversations with leadership. If your RTO is 4 hours but leadership thinks it’s 1, that’s a crisis waiting to happen.
• ▸Replication monitoring with alerting. Silent replication failure is the failure mode that causes data loss. Alert on lag, not just on failure.

DR isn’t a document. It’s a discipline. And the organizations that treat it like a document are the ones you read about in the news.