Assessing Response to Superstorm SandyBusiness Continuity Planning Put to the Test
In New York, New Jersey and other northeastern states, millions of residents are still coping with the devastation caused by Superstorm Sandy. Meanwhile, information security professionals are continuing to help carry out business continuity plans.
At least 76 people in the U.S. and two in Canada died during the storm, raising Sandy's overall death toll to 145 after earlier claiming 67 lives in the Caribbean, according to a midday Nov. 1 CNN report.
Millions were still without electricity on Nov. 1, and utilities in the hardest hit states, such as New Jersey and New York, said it could take a week to 10 days to restore power in some areas.
Estimates of the costs from Superstorm Sandy continued to rise. The risk assessment consultancy EQECAT pegs total economic damage from Sandy to be between $30 billion and $50 billion, with insurers picking up between $10 billion and $20 billion of that tab.
At the height of Sandy's violent run, many computer systems in key sectors, such as government, banking and healthcare, continued to function, in part, because of careful business continuity planning.
Despite dealing with major challenges, none of the officials at governments, banks and hospitals contacted by Information Security Media Group's editors reported significant problems with executing their disaster plans. But many say they'll conduct fuller assessments once the situation stabilizes.
"During the process and after the storm, we will always look for ways to improve preparedness for incidents and business continuity," says Neil Brazil of HSBC Bank USA.
Trouble with cellular telephone service, as well as power outages, meant those attempting to work at home - or reach out for assistance - faced serious challenges.
Verizon, a major telecom provider for the Northeast, managed to keep its national and regional command and control centers operational. But power outages and flooding took a toll on cellular service in some areas, such as lower Manhattan, resulting in tens of thousands of customers seeking places to find a strong signal to make calls - as well as sites where they could recharge their phone batteries.
"Although we will be working with all available resources to restore service for our customers, some pockets of damage are extensive and could take up to a week or more to fully restore," says Bob Mudge, president of Verizon's consumer and mass business division. "Some restorals will require physical rebuilding of our facilities, and others will require the return of commercial power."
Banking, Healthcare Issues
Power outages caused by Sandy's hurricane-force winds and floods forced hundreds of bank branches to close and shutter their automated teller machines. Wells Fargo's Sara Hawkins says the bank deployed three mobile ATMs to stricken communities.
Bank of America had hoped to deploy mobile ATMs in blacked-out lower Manhattan on Halloween night. "We ultimately couldn't get a signal in Manhattan," spokesman Mark Pipitone says. But, he says, BofA had planned to send a mobile ATM to Island Beach State Park along the New Jersey shoreline on Nov. 1 and hopes to have additional mobile ATMs in place in parts of northern New Jersey and Long Island, N.Y., on Nov. 2.
BofA's business continuity plan includes staging mobile ATMs in areas that lost power following a natural disaster, so why did the bank wait for several days to deploy them after Sandy? "We need time to assess the situation," Pipitone explains. "This effort works in conjunction with our overall business continuity planning, which includes getting traditional banking centers and ATMs back online."
With some of the largest hospitals in New York and New Jersey evacuating hundreds of patients because of power outages, Health and Human Services Secretary Kathleen Sebelius declared a public health emergency for both states.
The declaration means HHS may permit affected healthcare facilities to adjust certain operating procedures temporarily so services can be delivered. For example, it could enable qualified residents to be admitted to a nursing home and be covered by Medicare without the normal three-day prior hospital stay. It could also enable temporary alternate service locations to be quickly established.
Disaster Recovery Insights
In interviews with Information Security Media Group editors, security professionals in the healthcare, government and banking sectors offered assessments of their disaster recovery efforts.
The chief information security officer of an upstate New York network of hospitals affected by the storm says the biggest challenge was dealing with its business partners. "Our vulnerability is really the areas where we need to depend on service providers," says the CISO, who requested to remain unidentified. "Their performance is outside our control - power companies, ISPs, telecom companies. There is only so much you can do with back-up generators and redundant cables."
The CISO says he prepared for the storm by meeting with hospital administration, staff and vendors beforehand. "We made sure that all key personnel had multiple ways to get a hold of each other, and we made sure that every department had assigned primary contacts for each type of issue," the CISO says. "IT administration held regular daily conference calls throughout the storm."
Based on the smooth execution of the disaster recovery plan, the organization likely won't need to update the plan after the storm, the CISO says. "I don't think we could have improved on anything."
The CISO of a major New York City academic medical center also reports everything worked nearly as planned. The only problem was staffing: Only 2 percent of the IT staff could make it to the hospital on Oct. 29 and 30, although about half made it to their desks Oct. 31.
"Staff that is here, including myself, is working around the clock to ensure that there is appropriate coverage," says the CISO, who asked to remain anonymous. "We planned ahead and reserved hotel rooms, allowing people who don't live in the area to get a few hours of sleep here and there. ... Once the subways come online and power is restored to most areas, we'll be in good shape."
In Pennsylvania, where the storm claimed at least a dozen lives and cut off electricity to 1.2 million customers, keeping key people within state government and its IT and IT security areas connected was a vital part of the government's business continuity plan.
As with any large-scale emergency or natural disaster, the state relied on frequent communications and situation assessments to understand the impact of the storm on its operations, says Dan Egan, a spokesman for state Chief Information Security Officer Erik Avakian. "By receiving regular updates from emergency management officials, information technology staff and continuity of operations managers within each agency, we were able to develop and update our operating picture as the storm unfolded," Egan says.
In the coming weeks, Avakian and his team will survey state agencies to gain a better understanding of what went well and where the state can improve. "There are lessons to be learned from every event, both big and small, and so this is an important component of our program," Egan says. "We will incorporate the feedback into an after action review and make changes, as necessary."
Connecticut Chief Information Officer Mark Raymond also says it's too early to identify lessons learned from Sand to improve its business continuity plan. "That remains to be seen; we haven't assembled our after-action thoughts, yet," he says.
Because Connecticut experienced two major storms last year, the state gained insight that helped it plan for Sandy, Raymond says. In addition, he says, the state communicated often to a wide audience: holding twice-a-day media briefings as well as reaching out to municipalities and state agencies to coordinate its response to the superstorm. Connecticut also fully document plans, including who and how to contact members of the IT and IT security staffs to offer mutual aid and assistance.
In Vermont, Kris Rowley, the state's CISO, says difficulties coping with last year's Tropical Storm Irene led to the creation of a checklist of steps the state government followed in preparation for Sandy. Those steps included:
- State IT and IT security officials reviewed the continuity of operation and disaster recovery plans with all essential employees;
- The governor held news conferences to let citizens know what was going on and what to do;
- The state website had up-to-date information on all emergency preparedness plans and emergency contact information for various services;
- Officials planned for boots-on-the-ground assistance from outside of Vermont in case recovery help was needed. Some of these plans were with companies from Canada;
- Vermont's emergency response center was up and running with conference calls to all state first responders.
"Vermont was very fortunate in that we did not sustain much damage; there were some high winds, loss of power and minimal flooding in low-lying areas," Rowley says. "However, nothing happened that could not be addressed quickly and efficiently. Our power companies had ample resources available to address power losses, and there were numerous teams out to take care of fallen trees. We did not sustain any damage that really tested our plans."
Rehearsing for disasters helped banking institutions, including Capital One Financial, cope with Sandy.
"Each year, we rehearse and train for business continuity challenges that may be presented by floods, hurricanes, ice storms, tornadoes and other potential natural disasters or other emergency events," spokeswoman Amanda Landers says.
Brazil of HSBC Bank USA says securing employees and company facilities had been the priority for the early part of the week.
"We have long-standing policies and practices in place to advise and update our employees during emergency situations," he says. "Where possible, we have encouraged employees in the areas affected by the hurricane to work from home; we have significant capabilities to do so. We also have solutions to manage business activities when employees are affected by power outages."(Marianne Kolbasuk McGee, Tracy Kitten and Howard Anderson contributed to this story).