We have resolved the issue. We will be providing an RCA via our support portal in the next 2 business days. During this incident, the stats for memory and messages for Cloudhub runtime 3.9.x+ were not processed in real-time. If there are gaps in your stats for memory and messages, you need to contact support by July 27th.
Posted Jul 24, 2018 - 17:27 PDT
Monitoring
The engineering teams have completed the work, and we are processing message and memory stats in real time for runtimes workers 3.9.x+. There is a current backlog of messages that will take time to be processed. We expect the majority of messages to be processed within the next 12 hours.
We are moving this incident to Monitoring as we track there recover of the queues.
Posted Jul 23, 2018 - 23:44 PDT
Identified
We are currently seeing the delays with message and memory stats getting smaller after a fix that was put in place by our engineering teams. We are presently monitoring the queues, and they are being drained.
The engineering teams are working on bringing message and memory stats back to real-time while we process the backlog of messages.
We will have further status in the next hour.
Posted Jul 23, 2018 - 23:04 PDT
Update
We are still investing the stats ingest for runtimes workers 3.9.x+. Our engineerings are working on the causes for the delay and we should have an update posted shortly.
Posted Jul 23, 2018 - 09:56 PDT
Investigating
We are experiencing a degraded performance with Worker Cloud memory and message stats ingest. This issue only affects memory and message stats for runtimes 3.9.x+ and no transaction processed by the runtime is affected. Our engineers are investigating.
Posted Jul 23, 2018 - 01:06 PDT
This incident affected: Runtime Services (Runtime Fabric).