EU cluster Realtime recovered.

EU cluster frontend recovered.

EU cluster frontend recovered.

Two short periods of degraded service were experienced today between 15:51 and 15:57 CET and 16:13 and 16:21 CET. API, front-end, and webhook response rates were slow during this time and and some customers may have experienced time outs.

No data loss resulted.

The cause was due to flows abusing the storage (Maester) and queueing service limits. These flows have been suspended and we are working with the client to optimise their flows.

All services are restored and working normally....

EU cluster frontend recovered.

EU cluster frontend recovered.

EU cluster frontend recovered.

Post-mortem

Issue

On Friday 20th Jan, 2023 a significant spike in data volume was observed, this spike was also accompanied by an increase in data package size. The following increase in data processing times, increased message queue processing times and data volume combined to overwhelm our Kubernetes master node.
At 16:02 CET, the platform service became unavailable.

Google support where unable to correct this and so it was necessary to recreate the platform. During the downt...

Most of the flows have started. We are monitoring the situation.

Flow sample retrieval is not working

Issue

The recent issues experienced in the platform are because of our core scheduling service becoming temporarily overwhelmed because of a high volume of messages combined with many errors generated by integration flows transporting large batches of data.

Issues experienced included:
* Slow and failing sample retrieval
* Slow processing of message queues

Status/Resolution

The issues are now resolved, and no data loss is observed ...