We have identified the root cause. Optimization loads of some customer affected the entire cluster. We have resolved the infrastructure load balancing.

Because of higher load this afternoon, some jobs took more than 1 hour in the queue and we were unable to process them all. If more than 1 hour of queueing then we do no longer process the job.