Resolved
Jan 23 at 11:21am CET
Post-mortem
Issue
On Friday 20th Jan, 2023 a significant spike in data volume was observed, this spike was also accompanied by an increase in data package size. The following increase in data processing times, increased message queue processing times and data volume combined to overwhelm our Kubernetes master node.
At 16:02 CET, the platform service became unavailable.
Google support where unable to correct this and so it was necessary to recreate the platform. During the downt...