Service Disruption 9/30
Service Disruption 9/30
We are investigating an issue that caused a platform disruption overnight. We have addressed the issue, but will continue to monitor for a while longer. We will provide an update later today.
A release late yesterday was identified as causing the issue. It was rolled back this morning. Customers are no longer impacted since 9:07am CST.
Root cause was a logging update that exhausted network connections under load, blocking new connections and surfacing errors in action processing or cache writing.
We moved quickly to roll back and have also made root cause fixes to ensure connection usage will stay at the expected/original limits moving forward.
We continue to monitor and will provide an update later today.
Platform metrics are back within normal ranges since this morning. This incident has been resolved.