How We Almost Lost Our Production Kubernetes Cluster to a Misconfigured CronJob

We ran a 15-node Kubernetes cluster on GKE for our payment processing platform. The team was relatively new to Kubernetes — we had migrated from Heroku 6 months prior. We had basic monitoring with Prometheus and Grafana but our alerting coverage had gaps, especially around resource utilization at the node level. A developer had set up a CronJob to clean up stale payment session data from Redis.

The CronJob was configured to run every minute instead of every hour (*/1 * * * * vs 0 * * * *). Each job spawned a pod that pulled a 2GB Docker image (the dev had used the full application image instead of a slim utility image). The CronJob had no concurrencyPolicy set, so Kubernetes was spawning a new pod every minute without waiting for the previous one to finish. Over a weekend, this created hundreds of pending pods that consumed all schedulable resources. By Monday morning, new deployments couldnt schedule pods and existing pods were getting OOMKilled from node memory pressure.

An engineer noticed the issue during Monday morning standup when a deploy was stuck in Pending. We identified 847 completed and pending CronJob pods consuming cluster resources. We deleted the CronJob, cleaned up the completed pods, and the cluster recovered within 20 minutes. We then rewrote the CronJob with concurrencyPolicy: Forbid, proper resource limits, a slim Alpine-based image, and the correct schedule. We added alerts for unusual pod counts and node resource pressure.