The chronicles of Mizuho’s system failures

How badly do you need to mis-manage for your regulator to come to the following conclusion as to the root cause of your troubles?

• Disregard for system-related risks and expertise
• Disregard for the actual situation of the IT section
• Lack of sensitivity to the impact on customers and disregard for the actual situation of the sales section
• Attitude to refrain from saying what needs to be said and to only do what was instructed

In Mizuho’s case, there were eight well-chronicled and very public system outages from February through September 2021 that led the Financial Services Agency (FSA) to this scathing judgement.

Administrative Actions were originally expected in September, but then got postponed until November to include the more recent failures. Unfortunately, for the bank, the misery did not end there, and while it is preparing to submit its business improvement plan by Monday, January 17, 2022, further system issues surfaced.

1. February 28, 2021

Regarding the system failure on February 28, it was found that the Bank conducted data migration without sufficiently considering the risk of conducting such operation at the end of the month, when the system generally becomes overloaded, and resulted in causing suspension of services at a number of ATMs. Cash cards and passbooks were swallowed by ATMs and a number of customers were forced to keep waiting for days until the ATMs became operational again (see the articles headline picture).

2. March 3, 2021

On March 3, 2021, a network card in network equipment at a data center failed, and this problem destabilized communications for three minutes until the switch to a network card in a different system. Impacts during this period were (1) retention of inserted bankbooks and cards at ATMs (29 cases) and (2) partial failure in Numbers Lottery ticket purchase transactions via ATM or Mizuho Direct service (7 cases).

3. March 7, 2021

On March 7, 2021, errors occurred in centralized entry processing related to consolidated account time deposits for close to eight hours because of a mistake involving the omission of a system initialization process following a software update.

Customers who experienced failed time deposit transactions via ATMs or Mizuho Direct service received notification of the system failure and failed transaction and an explanation of the circumstances. Some time deposit services at ATM were temporarily suspended with the aim of preventing ATMs from retaining bankbooks and cards.

4. March 12, 2021

At 11:39pm on March 11, failure of communications control equipment within the core banking system and associated outages took close to seven hours to fix. Sufficient recovery procedures were not conducted for the foreign exchange system and processing was not completed by the specified deadline, resulting in 263 transfers to other domestic banks missing the deadline for March 12, and 761 foreign currency transfers did not complete deposit notification processing during the same day.

5. August 19–20, 2021

The fifth system glitch disrupted over-the-counter services at its branches throughout Japan, because the hardware that connects the core banking system to devices at bank offices broke down. Recovery was delayed as it took some time for Fujitsu Ltd., responsible for the system’s maintenance, to grasp the issue.

In an affront to customer service, Mizuho Bank did not give notice to customers until only 30 minutes before the start of business hours, with many clients wishing to transact in person already on the way to the bank.

6. August 23, 2021

Network errors at Mizuho Bank’s main data center resulted in large volumes of data logs being output, overloading the system processor and leading to a portion of ATMs and front-office terminals to become temporarily inoperable.

7. September 8, 2021

A malfunctioning disk component in the backup system’s disk device for the shared transaction platform led to a temporary interruption in network communications with other systems, resulting in a temporary suspension of a portion of ATMs among other issues.

8. September 30, 2021

In the restoration process after the system failure on September 30, a problem was found in relation to the Bank’s compliance with laws and regulations on asset freezing and other economic sanctions, as well as the Guidelines for Anti-Money Laundering and Combating the Financing of Terrorism.

The Financial Services Agency’s (FSA’s) Administrative Action issued on November 26, 2021, covered the above system failures, and requires a business improvement plan to be submitted by Monday, January 17, 2022. Unfortunately for Mizuho, their system troubles continued into late 2021 and early 2022, as per the below.

9. December 30, 2021

Money transfers through Mizuho Bank ATMs and its online banking service to other banks became unavailable for about one hour from around 3:30pm. The number of affected transactions was not disclosed.

10. January 11, 2022

The bank says its corporate clients had trouble logging into its online e-banking platform for over 3-and-a-half hours in the morning. The service handles money transfers, overseas remittances and foreign-exchange transactions.

This article is part of our Tokyo FinTech Publication, please follow us to read more from our writers, like hundreds of readers do every day. Please also register for our short weekly digest, published every Saturday, at the link below.

Should you live in Tokyo, or just pass through, please also join our Tokyo FinTech Meetup. In any case, our YouTube channel and LinkedIn page are there for you as well.