Resolved -
As of 04:07 Pacific, the impact to API Manager is resolved. API Gateway is now able to retrieve and apply all the configured policies.
API Gateway might continue to show errors for the application of previous versions of the policy but this has no impact on the gateway and the errors will disappear with a restart.
Mar 5, 04:39 PST
Monitoring -
After the actions taken by our technical teams, we are seeing a steady decrease in the error rate and expect to be fully out of impact within the next hour.
Mar 5, 04:01 PST
Update -
We have validated a fix for this incident in our lower environments and are working to deploy it now. Once deployed and we see the error rates dropping, we will update to monitoring.
Mar 4, 23:28 PST
Update -
This is a retroactive note at 7:27 PM Pacific time, we noted an inaccuracy in our original posting where we explained that existing applications without policy updates would continue to operate normally. This turned out to be incorrect. A status post was made at that time on our EU Status site, but it was not communicated here on the US Status site. Please refer to the previous message for remediation steps while we work to resolve the overall situation.
Mar 4, 21:03 PST
Identified -
We have addressed the backend issue causing the errors that have generated this incident, while we work on restoring the service fully, we have identified a remediation step that can be taken to restore your applications to their expected performance.
If your Mule Gateway is currently affected, proceed to restart your Mule Runtime.
If you are unsure if your gateway is currently affected, you can review the logs to see if there is any occurrence of an error message like:
Error deploying policy to application
com.mulesoft.mule.runtime.gw.policies.template.exception.PolicyTemplateAssetException: Unexpected error when requesting template metadata for
If you see occurrences of the above error message then you should proceed to restart your Mule Runtime.
Mar 4, 20:37 PST
Update -
Existing applications with the policies will continue to operate smoothly. However, in the event of application restarts or when deploying new apps, there is a risk that they may encounter difficulties in downloading and applying API Manager policies, resulting in deployment failures.
Mar 4, 16:14 PST
Update -
We are continuing to investigate this issue.
Mar 4, 16:06 PST
Investigating -
Customers may experience errors while using applying new policies to existing and new applications. Our teams are investigating.
Mar 4, 14:56 PST