We have resolved the issue. We will be providing an RCA via our support portal in the next 2 business days. During this incident, applications and API Gateways continued to run without interruptions.
Posted May 30, 2018 - 15:36 PDT
Monitoring
The services are back online. We are investigating the consequences of the service outage and monitoring recovery.
Posted May 30, 2018 - 15:06 PDT
Update
We are moving Analytics to Operational State and will be monitoring this service for the next hour. We are currently in the last phases of validation around Object Store and will move this entire incident to Monitoring shortly,
Posted May 30, 2018 - 12:00 PDT
Update
We have changed the status for Analytics from Partial Outage to Degraded Performance. Both Analytics and Object Store are on the path to recovery and we will post in 30 minutes further status.
Posted May 30, 2018 - 11:39 PDT
Update
The engineering teams are still working on the affected systems for both Analytics and Object Store. We have seen an improvement in both services and are still working on the final resolution. We will have another update in 1 hour, which is pending an additional set of changes.
Posted May 30, 2018 - 08:38 PDT
Update
Enhanced Logging service has returned to normal. We continue to work on restoring full functionality for Analytics and Object Store. We will have an update posted in 30 minutes.
Posted May 30, 2018 - 07:58 PDT
Update
We are also investigating issues with Enhanced logging, We will be posting an update shortly as the team is currently investigating.
Posted May 30, 2018 - 05:30 PDT
Identified
We identified additional errors on API Analytics data ingest and experience increased error rate on Object Store. The engineering team is working on resolving the issues.
Posted May 30, 2018 - 02:55 PDT
Monitoring
Insights service is back online and processing events in normal capacity. API Analytics is online but still processing queued events. Object Store continues to work normally. We are monitoring recovery.
Posted May 29, 2018 - 21:08 PDT
Update
Object Store Service has been restored. API Analytics Service has been restored, there are still some delays on processing the queued data . Insights Analytics still has high error rates.
Posted May 29, 2018 - 18:56 PDT
Update
API Analytics service has been restored and it's catching up with the data ingestion. We are still working on Insights Data ingestion. It also affected the usage of the Object Store in CloudHub, causing elevated errors on Object Store API.
Posted May 29, 2018 - 16:21 PDT
Identified
We have confirmed elevated error rate with Insights and API Analytics data ingress and have identified the issue and expect to have it resolved soon.
Posted May 29, 2018 - 14:40 PDT
Investigating
We are experiencing elevated error rates with Insights and API Analytics data ingress. Our engineers are investigating.
Posted May 29, 2018 - 14:14 PDT
This incident affected: Object Store v2 (Object Store v2 - Stats - us-east-1), Anypoint Management Center (API Analytics), and Runtime Services (Insights).