HSBC struggles with customer transactions after crash of computer system
Many transactions not recorded
HSBC Bank USA is still scrambling to recover after its core banking computer system mysteriously crashed Sunday, preventing it from recording withdrawals and deposits, and wreaking havoc on consumers and businesses nationwide.
Bank officials acknowledged a serious problem Tuesday evening, saying the computer crash “impacted our ability to process customer transactions,” according to a customer statement provided to The Buffalo News.
That forced Western New York’s largest bank to “immediately” activate its emergency plan to “restore files and recover our systems.”
In the meantime, daily and overnight processing was delayed, and many transactions were not posted to accounts, leaving customers and bank personnel unsure of exactly how much money anyone had.
Officials said in the statement Tuesday that most of the affected systems are back up and running, but “we are still working through the remaining recovery work.” No data, transactions or money was actually lost.
“Our priority is to resume our normal service levels as quickly as possible and to ensure that the backlog of transactions is processed accurately and efficiently,” the bank said. “This we absolutely commit to do.”
But a spokeswoman admitted the bank does not know how long that will take or whether problems will continue today.
“We sincerely apologize for any inconvenience this may have caused,” the bank said. “We very much appreciate your business and thank you for your continued patience.”
The Delaware-based bank, formerly Marine Midland Bank, has been severely hampered by technical problems for three days, preventing customers from accessing accounts and money. Officials “managed” certain transactions manually at branches and call centers to “minimize interruption of customer service,” the bank said.
HSBC, a unit of London-based HSBC Holdings Plc, the world’s No. 3 bank by value, has the most branches in the state, with 389 offices from Western New York to the Hamptons on Long Island. It has another 67 offices in eight other states, mostly in California, Florida and New Jersey. All were affected.
HSBC’s online banking system appeared to come back up Tuesday after being down Monday, but customers said their balances were not updated. Meanwhile, they had mixed success withdrawing money, especially any large amounts, as customers said the bank had trouble verifying how much money they had in their accounts.
Scott Hahn, of Lancaster, tried for two days to cash a check at the Depew branch so he would have money for a home improvement project and school shopping without using credit cards.
Tuesday, branch staff could look up his account and allow a partial withdrawal, but he couldn’t get the full amount of his out-of- state check because they couldn’t verify if he had enough money in his account to cover it. He was told he would have to wait until the check clears.
“It was pretty clear they were still having some problems,” said Hahn, 32.
The bank initially downplayed the issue, calling it an “infrastructure problem” or a “systems issue involving disks in our mainframe computer” that affected “certain applications.” In reality, it was much worse.
“It’s very serious to have your core system go down,” said Bart Narter, senior vice president of the banking group at technology research firm Celent. “It is a major technical failure.”
A bank’s core system is the back-office computer that keeps track of checking and other account balances and transactions. Many other bank systems, such as online banking and ATMs, rely on the information in that system. Other computer systems are unrelated.
“The core system is the system of record, the absolute arbiter of how much money you have and how much money you deposited today,” Narter said.
Most U. S. banks do not operate in “real time,” but rather keep computerized notes or “memos” of all transactions that occur during a given day, Narter said. At night, during a specified window when the bank shuts down, officials run the core program to update the accounts using “batches” of those transaction records. They then bring the system back up in the morning.
If the hard drive fails, the bank has to reboot and restore the system, and then rerun the overnight batches to catch up. That takes time at a big bank.
“This is a bank’s worst nightmare, to have the core banking system goes down,” he said. “That means any channel you go to, they’re running blind.”
Narter said he has only seen this happen one other time in the last five years, and that was six months ago at San Franciscobased Wells Fargo & Co. In that situation, however, the crisis occurred entirely over a weekend, so it was mostly restored by the start of business Monday, and there was less impact.
HSBC officials now plan to spend the next several weeks figuring out what happened. “We are reviewing what transpired in this incident in order to identify root causes and are making every effort to prevent a similar incident from occurring in the future,” the bank said.
Customers are angry. “This problem is a lot worse than what they’re saying,” said Al Groccia, manager of a family-owned ATM firm in New York City, which needs cash to stock its machines. “I’m just furious. I need the funds to operate.”
Groccia’s wife owns Cash To Go ATM LLC, an independent operator of 20 ATMs in convenience stores and restaurants in New York City. HSBC advanced them some money Monday but wouldn’t advance more without knowing how much the business had in its account.
He could log into online banking, but couldn’t see an updated balance. And although money was transferred into the account, he said HSBC couldn’t verify it was there. “The money is being transferred to HSBC, but they don’t know where it is,” he said. “They don’t even see the money showing up.”
As a result, some of his ATMs were down, costing the company both money and good will, as the stores and restaurants that counted on the ATMs to help them with sales are also losing business. “It’s such a terrible snowball effect,” he said.
“I was told they don’t know when the system will be up and running. I don’t even know if I’m going to have funds for [today].”
