Tokyo FinTech
Published in

Tokyo FinTech

The chronicles of Mizuho’s system failures

How badly do you need to mis-manage for your regulator to come to the following conclusion as to the root cause of your troubles?

  • Disregard for system-related risks and expertise
  • Disregard for the actual situation of the IT section
  • Lack of sensitivity to the impact on customers and disregard for the actual situation of the sales section
  • Attitude to refrain from saying what needs to be said and to only do what was instructed

In Mizuho’s case, there were eight well-chronicled and very public system outages from February through September 2021 that led the Financial Services Agency (FSA) to this scathing judgement.

Administrative Actions were originally expected in September, but then got postponed until November to include the more recent failures. Unfortunately, for the bank, the misery did not end there, and while it is preparing to submit its business improvement plan by Monday, January 17, 2022, further system issues surfaced.

1. February 28, 2021

Regarding the system failure on February 28, it was found that the Bank conducted data migration without sufficiently considering the risk of conducting such operation at the end of the month, when the system generally becomes overloaded, and resulted in causing suspension of services at a number of ATMs. Cash cards and passbooks were swallowed by ATMs and a number of customers were forced to keep waiting for days until the ATMs became operational again (see the articles headline picture).

2. March 3, 2021

On March 3, 2021, a network card in network equipment at a data center failed, and this problem destabilized communications for three minutes until the switch to a network card in a different system. Impacts during this period were (1) retention of inserted bankbooks and cards at ATMs (29 cases) and (2) partial failure in Numbers Lottery ticket purchase transactions via ATM or Mizuho Direct service (7 cases).

3. March 7, 2021

On March 7, 2021, errors occurred in centralized entry processing related to consolidated account time deposits for close to eight hours because of a mistake involving the omission of a system initialization process following a software update.

Customers who experienced failed time deposit transactions via ATMs or Mizuho Direct service received notification of the system failure and failed transaction and an explanation of the circumstances. Some time deposit services at ATM were temporarily suspended with the aim of preventing ATMs from retaining bankbooks and cards.

4. March 12, 2021

At 11:39pm on March 11, failure of communications control equipment within the core banking system and associated outages took close to seven hours to fix. Sufficient recovery procedures were not conducted for the foreign exchange system and processing was not completed by the specified deadline, resulting in 263 transfers to other domestic banks missing the deadline for March 12, and 761 foreign currency transfers did not complete deposit notification processing during the same day.

5. August 19–20, 2021

The fifth system glitch disrupted over-the-counter services at its branches throughout Japan, because the hardware that connects the core banking system to devices at bank offices broke down. Recovery was delayed as it took some time for Fujitsu Ltd., responsible for the system’s maintenance, to grasp the issue.

In an affront to customer service, Mizuho Bank did not give notice to customers until only 30 minutes before the start of business hours, with many clients wishing to transact in person already on the way to the bank.

6. August 23, 2021

Network errors at Mizuho Bank’s main data center resulted in large volumes of data logs being output, overloading the system processor and leading to a portion of ATMs and front-office terminals to become temporarily inoperable.

7. September 8, 2021

A malfunctioning disk component in the backup system’s disk device for the shared transaction platform led to a temporary interruption in network communications with other systems, resulting in a temporary suspension of a portion of ATMs among other issues.

8. September 30, 2021

In the restoration process after the system failure on September 30, a problem was found in relation to the Bank’s compliance with laws and regulations on asset freezing and other economic sanctions, as well as the Guidelines for Anti-Money Laundering and Combating the Financing of Terrorism.

The Financial Services Agency’s (FSA’s) Administrative Action issued on November 26, 2021, covered the above system failures, and requires a business improvement plan to be submitted by Monday, January 17, 2022. Unfortunately for Mizuho, their system troubles continued into late 2021 and early 2022, as per the below.

9. December 30, 2021

Money transfers through Mizuho Bank ATMs and its online banking service to other banks became unavailable for about one hour from around 3:30pm. The number of affected transactions was not disclosed.

10. January 11, 2022

The bank says its corporate clients had trouble logging into its online e-banking platform for over 3-and-a-half hours in the morning. The service handles money transfers, overseas remittances and foreign-exchange transactions.

This article is part of our Tokyo FinTech Publication, please follow us to read more from our writers, like hundreds of readers do every day. Please also register for our short weekly digest, published every Saturday, at the link below.

Should you live in Tokyo, or just pass through, please also join our Tokyo FinTech Meetup. In any case, our YouTube channel and LinkedIn page are there for you as well.

--

--

--

一般社団法人 (General Incorporated Association) Tokyo FinTech is registered as a non-profit organization in Japan, promoting the domestic ecosystem through innovation

Recommended from Medium

Shield Finance + Paralink Partnership

Trump’s Website Hacked, Scam Asked Crypto Owners to Decide Fate of ‘Classified Information’

SOOOOO THIS IS A TEst tun.

{UPDATE} AimTrainer Hack Free Resources Generator

Why Do Hackers Bother with Small Sites?

Release Notes: January https://t.co/0UcvnVv6Oj #cutomerservice https://t.co/o4lMxEPKxD

DeFi in Data Economy: Decentralized Data Exchange based on privacy-preserving computation network

Introduction to Ethernet

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Norbert Gehrke

Norbert Gehrke

Passionate about strategy & innovation across Asia. At home in Japan. Connector of people & ideas.

More from Medium

How fund managers can keep FX costs down

How to choose your startup co-founder titles? | CEO, CFO, COO etc.

HIPAA Compliance in the Cloud

From Failure To Success: 2 Concepts From Silicon Valley #5