An AWS Outage Disrupts Services at Financial Institutions
On Monday, a failure in one of Amazon Web Services’ (AWS) internal subsystems in its northern Virginia data center led to errors and latencies that rippled across the internet, affecting various services, including those of several banks and fintech companies. AWS, the world’s largest cloud provider, is integral to many online services, making its reliability crucial to a vast array of industries, including the financial sector.
This incident underscores the deep-seated reliance of the financial sector on centralized cloud infrastructure. The situation is reminiscent of extensive outages last year, which were triggered by a defective update at cybersecurity firm CrowdStrike. AWS kept its users updated on the progress of the investigation and recovery from the outage on its status webpage. AWS has not indicated that a cybersecurity incident was the cause of the disruptions.
The AWS Outage: A Closer Look
According to AWS, engineers began investigating heightened error rates and latencies in multiple AWS services in the us-east-1 region (the northern Virginia AWS data center) at 3:11 a.m. on the East Coast on Oct. 20. The company later determined that an “underlying internal subsystem,” which monitors the health of network load balancers, was the cause of the network connectivity issues.
Several critical AWS services, including S3 (data storage), CloudFront (web content delivery), RDS (database service), DynamoDB (database service), Lambda (event-driven code execution), and SNS (notification service) were affected. Certain EC2 instances, which are virtual servers in Amazon’s Elastic Compute Cloud, also encountered issues when trying to boot up. Mitigations were reported to be progressing at 2:22 p.m., with decreasing network connectivity issues and an increase in the launches of new EC2 instances.
Impact on Banks and Credit Unions
Downdetector, a service that aggregates crowd-sourced reports of service outages, recorded issues at numerous financial companies and related platforms on Monday. Navy Federal Credit Union, Truist, Ally, Chime, and Venmo all experienced spikes in user reports of outages. A spokesperson for Navy Federal acknowledged the nationwide outage and its impact on multiple member service platforms, without specifying which services were particularly affected.
According to the spokesperson, the credit union’s support teams quickly identified the national outage and started working on service restoration while engaging with external vendors to determine the cause. Truist, Ally, Chime, and Venmo did not respond to requests for comment immediately.
Consequences for Payment Processing
Payment processing organizations faced immediate downstream effects due to the outage, according to Monica Eaton, founder and CEO of Chargebacks911, a company that assists merchants with charge-back fraud. “When AWS sneezes, half the internet catches the flu,” she said. Eaton predicted a surge in claims of duplicate charges or unfulfilled services, attributing most of these to confusion rather than fraud. She advised businesses to proactively check for duplicate charges, send advance notifications to affected users, document the outage window, and offer swift refunds where necessary.
Preparing for Future Failures
Cliff Steinhauer, director of information security and engagement at the National Cybersecurity Alliance, highlighted the incident as a demonstration of how reliant our lives have become on a few major cloud providers. He pointed out that even the biggest systems are not immune to outages, which often expose the fragility of interconnected systems when the internet’s infrastructure is so centralized.
Steinhauer further suggested that businesses could use this outage as a valuable learning experience to identify gaps in their business continuity plans and evaluate their backup solutions. He emphasized the importance of reviewing redundancy and disaster recovery strategies to ensure that critical operations are not tied to a single cloud region or provider.
Source: Here
