AWS investigating S3 problem at major datacenters location
Amazon's initial response to Tuesday's problems.
An issue with S3 cloud storage at Amazon's massive datacenter region in Northern Virginia on Tuesday caused problems for several AWS customers. After several hours, AWS said Tuesday afternoon that the problem was resolved.
"We are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally," Amazon Web Services said on its service health dashboard at 2:08 pm PT.
The cloud infrastructure provider did not immediately respond to inquiries from ZDNet regarding the details of the problem or AWS's response to it.
About three hours earlier, AWS notified customers via the dashboard: "We're continuing to work to remediate the availability issues for Amazon S3 in US-EAST-1. AWS services and customer applications depending on S3 will continue to experience high error rates as we are actively working to remediate the errors in Amazon S3." .
AWS noted on Twitter that the dashboard itself was impacted by the problem:
The US-East region is Amazon's oldest datacenters location, and Tuesday's issue -- which AWS did not identify as an outage -- impacted several companies including Medium, Slack, and Trello.
AWS experienced a similar incident in 2015, when an outage lasting several hours impacted multiple major companies.
Update: At 11:35 am PT, AWS noted that its dashboard has recovered and is delivering updates on individual services. Several services were impacted, according to the dashboard, including: Athena, Elastic MapReduce, Kinesis Firehose, Amazon Simple Email Service, AWS CloudFormation, EC2 Container Registry, Elastic Compute Cloud, Redshift, Elastic Beanstalk, and Lambda.
AWS also said they believe they've found the root cause, though problems continue:
"We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue."
Update: At 12:52 pm PT, AWS said: "We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour."
Update: At of 1:12 pm PT, AWS said: "S3 object retrieval, listing and deletion are fully recovered now. We are still working to recover normal operations for adding new objects to S3."
Update: At of 1:49 pm PT, AWS said: "We are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally."
VIDEO: Try these fun (and completely useless) Amazon Alexa commands