Improving Data Center Resiliency With Disaster Recovery Checklists

Since the beginning of 2020, people have been forced to reckon with the grim reality that we are all vulnerable, and crises can come in many unpredictable forms. Almost overnight, a global pandemic sent the world’s workforce home and stretched networks to their limits. We experienced a severe winter storm in Texas, of all places, that disabled the state’s power grid; wildfires that ravaged the western U.S., Australia, and South America; and other severe weather events around the globe. The human and financial toll of these disasters has been nothing short of devastating, and the effects will be felt for years, if not decades.

For data center owners, accepting any notion that the current infrastructure and its robust backup power systems are immune to these sorts of crises would be a huge mistake. Climate change is causing more and more extreme weather events each year, and the U.S. is shattering annual records for billion-dollar disasters. This summer will be the hottest on record for more than 30 cities across America. According to an Uptime Institute report, three in five respondents think there will be more IT service outages as a direct result of the impact of climate change, and nearly 90% think climate change will drive up the cost of data center infrastructure and operations over the next 10 years.

Beyond climate change, recent hacks that target national infrastructure have illustrated how vulnerable we are when these catastrophic cyberattacks occur. All of these unexpected crises are significant threats to data centers, and these situations can be made even worse by human error, lack of preparedness, and poor judgement. That’s why there’s never been a more appropriate time for organizations to put together a plan to secure their data centers in the event of an emergency. A good emergency plan should be a living document and a guide for ongoing planning, exercises, and updates. Below are some of the key factors to consider when assembling a plan.

• Ensure the data center is waterproof — If the threat is a hurricane, flooding, or something weather-related, take the necessary steps to harden your facility. Store loose items and make sure servers are secured in their racks. Check that gutters and storm drains are clear. Make sure doors can be sealed against high winds and blowing rain. Water is the enemy of the data center, so do everything necessary to keep it out.

• Backup data often — Many data centers conduct routine backups once a week. If you know severe weather is coming, increase the frequency of those backups. We can’t always know when a disaster is going to strike, so organizations should consider making daily backups a regular practice. Also, consider where data is being backed up. It should go off-site to asafe and secure location.

• Always check the generator — There is a tendency to set and forget a generator, but that piece of machinery requires maintenance and upkeep to ensure it performs as expected when needed. Is it full of clean fuel? Are the fuel line and air filter free of contaminants? Test the generator regularly and ahead of any anticipated weather events. Line up at least three vendors to deliver fuel in the event of an extended outage. Remember, fuel is often at a premium after a disaster, and yours will not be the only organization requiring delivery.

• Remember the edge sites — Today, the enterprise data center is just one piece in a distributed network. Many organizations manage multiple edge sites that are increasingly mission critical, and they must be considered in disaster planning. In many cases, the core data center may be safe from a specific event, but one or more edge sites could be at risk. Prioritize by criticality and have a plan for those facilities and their personnel.

• Mind the cloud — Just because some of your data and applications are housed in the cloud does not mean they are always safe from emergency events. Those cloud servers are in a data center somewhere, and you should know how your cloud provider will handle a potential disaster. How often are they backing up data? Do they have redundant sites? Ask these questions before a crisis, because once disaster strikes, it’s too late.

• Consider emergency staffing — In the event of a significant disaster, local employees may be unavailable to work. They may have evacuated with their families, be dealing with urgent damages to their homes or vehicles, or be unable to reach the data center due to impassable roads. Consider bringing in emergency crews and establishing crisis housing near the data center to ensure you have on-site personnel.

• Beware of any opportunists — Hackers see natural disasters or similar emergency events as an opportunity to access networks while attention is focused elsewhere. Make sure your information security and physical security teams are prepared for bad actors.

Circumstances and personnel change, equipment ages and is refreshed, and risks evolve over time. To be sure your data center is ready when disaster strikes, having a disaster recovery checklist is nonnegotiable.  

Kate Fulkert   

Kate Fulkert is director of global business continuity/disaster recovery for Vertiv. She earned a master’s degree in emergency and disaster management from Georgetown University, is a master business continuity professional, holds a crisis management certification from the Massachusetts Institute of Technology, and has completed certified information systems security professional training.