Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster Autoscaler gets OOMKilled instead of scaling down #7873

Open
MenD32 opened this issue Feb 26, 2025 · 0 comments
Open

Cluster Autoscaler gets OOMKilled instead of scaling down #7873

MenD32 opened this issue Feb 26, 2025 · 0 comments
Labels
area/cluster-autoscaler kind/bug Categorizes issue or PR as related to a bug.

Comments

@MenD32
Copy link

MenD32 commented Feb 26, 2025

Which component are you using?:
/area cluster-autoscaler

What version of the component are you using?:

v1.26.2

Component version:

What k8s version are you using (kubectl version)?:
1.30.0

kubectl version Output
$ kubectl version

What environment is this in?:
AWS

What did you expect to happen?:
Cluster autoscaler to scale down nodes when no longer needed

What happened instead?:
Cluster autoscaler was getting OOMKilled (We where using about ~340 nodes, with 0.6G memory limit).
When a new node of cluster autoscaler gets created, the node cooldown before scaling down gets back to 0s, Cluster Autoscaler was continuously getting OOMKilled after 1 loop and thus never managing to scale down nodes.

How to reproduce it (as minimally and precisely as possible):
Install Cluster Autoscaler with a memory limit, scale up nodes within the existing node groups until it gets OOMKilled .

Anything else we need to know?:

Getting OOMKilled from ~340 nodes with 0.6G ram limit is very surprising, but the thing that made this bug truly devastating, is that is was able to run 1 loop, which could've scaled down the nodes, but the timer had restarted, this makes me question the HA-ness of cluster autoscaler. I'd like to submit a fix where this data is added as an annotation to the node itself, thus making the deployment stateless, in that regard.

relevant configs are that the memory limit was 600mb, and that cooldown on down scaling was 10m.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cluster-autoscaler kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants