Solid contingency plans can avert disaster
Backup systems kept HSBC’s problems from being worse
HSBC Bank USA has struggled mightily to recover from the crash of a key computer system two weeks ago, but it could have been a lot worse if the banking giant hadn’t had a backup plan and system in place.
That’s why experts say it’s critically important for businesses of all sizes to develop contingency plans and back up their computer systems in case of a disaster or crisis of any kind. And the same applies for individuals with home computers.
Such preparation, including designation of a single coordinator, will ensure that a business can get back on its feet quickly and restore operations and service to customers with minimal or no interruption. It could even mean the difference between survival and failure.
“When you’re in a crisis situation or disaster situation, you need a crisis management plan in place,” said Paul Sullivan, vice president and general manager of Agility Recovery Solutions, a disaster recovery firm in Mississauga, Ont., that has clients in Western New York.
“If you’re not handling the crisis effectively, you will eventually get back up, but you will either have spent too much money, made errors in how you spent the money, and you will have taken a lot longer than what normally would have occurred.”
Similarly, he said, backing up a home computer can make the difference between an easy recovery of personal information, e-mail addresses and files, or enormous frustration and loss.
“You can almost equate it to losing your wallet,” Sullivan said. “Losing your desktop is the same kind of thing. If . . . it’s destroyed, lost or stolen, think about the effect it has on your life.”
Wholesale insurance brokerage H. R. Keller and Co. hadn’t updated its outdated contingency plan in more than 20 years, and it was focused on computer backups. So owner and president Eric P. Keller knew he had to upgrade his preparations.
The 44-year-old Tonawanda company and its 32 employees provide about 1,200 retail insurance agencies in New York and Pennsylvania with hard-to-place property and casualty insurance coverage, mostly for their business clients.
“If there is some major disaster, somebody in the insurance business needs to be there to respond,” Keller said. “We’ve got customers records here. We’ve got to be here.”
And while a lot of businesses have “business interruption” insurance coverage, Keller said that doesn’t replace planning. “Without any plan, your business interruption plan would take care of the costs, but it doesn’t figure out for you how to do it,” he said.
The issue of contingency plans gained attention here after HSBC’s core banking computer system went
down Aug. 15 because of a faulty disk in its data storage system. The bank spent the next 10 days repairing the problem, restoring data, and catching up on transactions so that customer accounts could be properly updated. Customers are still reporting sporadic problems, but most issues have since been resolved.
The bank’s problems were minimized compared to what could have happened. The computer crash affected only the posting of transactions that had just occurred or that were being processed at that time and over the coming days.
That’s because the bank had backup electronic records of all the accounts and prior transactions, enabling it to quickly restart using the most recent backup and begin the catch-up work from there. And the bank has a “robust contingency plan and invoked it when this issue arose,” enabling it to tackle the problem right away, said spokeswoman Linda Recupero.
“Contingency plans for all areas of our bank are vital not only to doing business, but to safe and expeditious recovery when an issue like this occurs,” she said.
In HSBC’s case, banks are required by their regulators to have such emergency plans and systems in place, so there was no choice about doing so for the British banking giant. And many other large businesses are sophisticated enough to make similar preparations.
But a lot of smaller businesses don’t, or they don’t fully consider and prepare for all the possible scenarios, leaving them vulnerable when a crisis hits. In particular, while many businesses today back up their computer files and data, many don’t store the backups safely.
“They’ll back up the data and put the tape on top of the server,” Sullivan said. “What they didn’t do is take the tape and store it somewhere away from that server, even outside that city.”
That became a particular issue for businesses in New Orleans that had backed up their data and stored it down the street or several blocks away. When Hurricane Katrina flooded the entire city, “they lost everything,” Sullivan said.
The same is true for individuals. Individuals today use their home computers or laptops for electronic banking, online shopping, storing personal information and documents.
But storing a backup of your home computer in a desk drawer won’t help if your home is destroyed. Instead, Sullivan says, ship it to a friend or relative, or use a free or low-cost service.
Emergency planning also means considering factors outside your control. For example, Agility has customers in Iowa whose buildings and property were undamaged by recent flooding. But their telecommunications and power providers were underwater or otherwise impaired, leaving the businesses unable to fully function.
“They had a backup plan, but they were reliant on the outside supply chain for their network and power,” Sullivan said.
And flooding can also occur from something more mundane, like a pipe bursting in a shared office building.
But such natural disasters and physical damage represents only about 20 percent of the problems businesses can experience, Sullivan said. The rest, especially telecommunications outages, are more routine or day-to-day in nature.
That’s why he urges businesses to talk to their service providers, such as the phone or utility company, and make sure they have their own emergency plans and test them.
And even if companies have a disaster recovery plan dealing with their data, Sullivan said less than half of businesses have a business continuity plan dealing with protecting their business operations and employees.
Companies like Agility help clients develop emergency backup plans, and also provide assistance during a crisis. Such arrangements can also be made without outside help.
That could include going to a backup “hot site” to operate from temporarily or bringing in a mobile trailer so employees don’t have to travel a long distance. It also includes being ready to bring in emergency generators, extra computer equipment and phones, as well as arranging for satellite or other communications setups if the regular office is not available.
Keller used Agility’s organizational package to map out how it would respond to a disaster, creating checklists and lists of equipment and licensing information so employees would know “exactly what needs to be done, every step of the way, in the event something happens.” Agility has one copy, and Keller stored two copies off-site in different places.
Keller also took Agility’s advice and contacted the town building department to get advance permission to bring in trailers in case “our building burned down one night.” Agility stores such temporary office trailers in Mississauga and Syracuse.
Those preparations paid off sooner than Keller would have expected. When the October Surprise storm struck in 2006, the agency lost power for more than three days and needed a generator. Agility drove in a diesel- powered trailer, and an electrician switched the main electrical panel from the public power system to the generator.
“We fired it up and there was enough power [to] run everything,” Keller said.
Even so, not everything worked as planned. Keller had prearranged with Noco to get diesel fuel for the generator. But Noco’s power was also out. Keller couldn’t even call them.
“I drove over to their depot because their phone system wasn’t working,” he said.
