[WIP] VPA: Implement in-place updates support #7673

maxcao13 · 2025-01-07T18:10:29Z

What type of PR is this?

WIP
Depends on: #7813

/kind feature
/kind api-change

What this PR does / why we need it:

This PR is an attempt to implement VPA in-place vertical scaling according to AEP-4016. It uses the VPA updater to actuate recommendations by sending resize patch requests to pods which allows in-place resize as enabled by the InPlacePodVerticalScaling feature flag in k8s 1.27.0 alpha and above (or by eventual graduation).

It includes some e2e tests currently according to the AEP, but I am sure we will probably need more.

This PR is a continuation of #6652 started by @jkyros with a cleaner git commit history.

Which issue(s) this PR fixes:

Fixes #4016

Special notes for your reviewer:

Notable general areas of concern:

~~See this comment about the potential need for a third containerResize policy that currently does not exist in the k8s api.~~
- We have amended the AEP since we cannot guarantee no container restarts. This is no longer a problem.
~~Needs more attention on how disruptive/disruption-free updates (should affect)/(are affected) by PodDisruptionBudgets.~~
- This can be covered in future enhancements.
We just kind of hacked the in-place stuff into the eviction limiter, maybe it should have been its own thing, or maybe we need a "disruption limiter", but in-place and eviction needed to know about each other because they have the same "disruption limit"
For now, there are many many TODOs literred throughout the code which need attention from reviewers/maintainers. A lot is because of design decisions I probably shouldn't make on my own. I resolved some of John's TODOs but he still has relevant comments that need to be addressed as well. I am using the TODOs as the "special notes for your reviewer" section, if people would like a comment somewhere which lays them all out nicely, I'm more than happy to make one.
- I've limited as many TODOS as possible, but I'm using the rest of the TODOs as review markers to draw attention to parts of the discussion that need community discussion.
Requires a lot more unit testing, but if a lot of architecture is to change, I chose delay writing them until I get feedback.
There's additional comments by John which can help aid review in the earlier commit descriptions.

Does this PR introduce a user-facing change?

In-place VPA scaling implemented, it can be enabled by setting `updateMode` on your VPA to `InPlaceOrRecreate` (Depends on `InPlaceVerticalPodScaling` feature gate being enabled or having graduated)

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[AEP] https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support
Depends on: 
[KEP] https://github.com/kubernetes/enhancements/tree/25e53c93e4730146e4ae2f22d0599124d52d02e7/keps/sig-node/1287-in-place-update-pod-resources

k8s-ci-robot · 2025-01-07T18:10:39Z

Hi @maxcao13. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-triage-robot · 2025-01-07T18:55:03Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

adrianmoisey · 2025-01-07T19:35:55Z

/ok-to-test

omerap12

This is just my first review after going through all the changes. I will go over it multiple times, but these are my initial comments for now.

vertical-pod-autoscaler/hack/run-e2e-locally.sh

vertical-pod-autoscaler/pkg/utils/annotations/vpa_inplace_update.go

vertical-pod-autoscaler/pkg/apis/autoscaling.k8s.io/v1/types.go

vertical-pod-autoscaler/pkg/updater/eviction/pods_eviction_restriction.go

vertical-pod-autoscaler/pkg/updater/priority/update_priority_calculator.go

maxcao13 · 2025-01-15T02:27:29Z

Appreciate the review! I will respond to the other comments tomorrow, just wanted to get the easy stuff out of the way and less cluttered.

omerap12 · 2025-01-15T12:55:19Z

Appreciate the review! I will respond to the other comments tomorrow, just wanted to get the easy stuff out of the way and less cluttered.

Sure, take your time

vertical-pod-autoscaler/pkg/updater/eviction/pods_eviction_restriction.go

maxcao13 · 2025-01-16T23:47:28Z

This is an important note/question. It may just be me being confused about the AEP details, but AFAIK there is no way for a user to guarantee that a resize request will be disruptonless. We can only guarantee that a resize will be disruptful (by configuring container resize restart policy). So how can we dictate the resizing type based on the conditions in the AEP? e.g. in the AEP it states

InPlaceOnly and InPlaceOrRecreate will attempt to apply a disruption-free update in place if it meets at least one of the following conditions:

Quick OOM,
* Outside recommended range,
* Significant change.

InPlaceOnly and InPlaceOrRecreate will attempt to apply updates that are not disruption-free in place under the same conditions that apply to updates in the Recreate mode.

Since the conditions are different, how can we actually make sure the first set of actions will actually be disruption free? I think this problem requires the need for a third resizePolicy MustNotRestart as mentioned in the AEP. But that does not exist right now. That will also help this test actually make sense.

maxcao13 · 2025-01-30T20:44:01Z

This will be affected by KEP-1287 update in 1.33: kubernetes/enhancements#5089

~~Additionally, we should probably consider feature gating this.~~

This just addes the UpdateModeInPlaceOrRecreate mode to the types so we can use it. I did not add InPlaceOnly, as that seemed contentious and it didn't seem like we had a good use case for it yet.

Signed-off-by: Max Cao <[email protected]>

So because of InPlacePodVerticalScaling, we can have a pod object whose resource spec is correct, but whose status is not, because that pod may have been updated in-place after the original admission. This would have been ignored until now because "the spec looks correct", but we need to take the status into account as well if a resize is in progress. This commit: - takes status resources into account for pods/containers that are being in-place resized - makes sure that any pods that are "stuck" in-place updating (i.e. the node doesn't have enough resources either temporarily or permanently) will still show up in the list as having "wrong" resources so they can still get queued for eviction and be re-assigned to nodes that do have enough resources

This commit makes the eviction restrictor in-place update aware. While this possibly could be a separate restrictor or refactored into a shared "disruption restrictor", I chose not to do that at this time. I don't think eviction/in-place update can be completely separate as they can both cause disruption (albeit in-place less so) -- they both need to factor in the total disruption -- so I just hacked the in-place update functions into the existing evictor and added some additional counters for disruption tracking. While we have the pod lifecycle to look at to figure out "where we are" in eviction, we don't have that luxury with in-place, so that's why we need the additional "IsInPlaceUpdating" helper.

The updater logic wasn't in-place aware, so I tried to make it so. The thought here is that we try to in-place update if we can, if we can't or if it gets stuck/can't satisfy the recommendation, then we fall back to eviction. I tried to keep the "blast radius" small by stuffing the in-place logic in its own function and then falling back to eviction if it's not possible. It would be nice if we had some sort of "can the node support an in-place resize with the current recommendation" but that seemed like a whole other can of worms and math.

sftim · 2025-02-11T23:46:50Z

vertical-pod-autoscaler/pkg/utils/annotations/vpa_inplace_update.go

+const (
+	// VpaInPlaceUpdatedLabel is a label used by the vpa inplace updated annotation.
+	VpaInPlaceUpdatedLabel = "vpaInPlaceUpdated"
+)


Is this a label or an annotation?

I copied the style from here:

autoscaler/vertical-pod-autoscaler/pkg/utils/annotations/vpa_observed_containers.go

Line 29 in 54fe60e

VpaObservedContainersLabel = "vpaObservedContainers"

. I'm not really sure of the intention of using the word label since it was written 6 years ago, but it probably makes more sense to be VpaInPlaceUpdatedAnnotationKey or something.

sftim

There are quite a few TODO comments. Is this PR ready for merge?

sftim · 2025-02-11T23:48:26Z

For now:
/retitle [WIP] VPA: Implement in-place updates support

You can change the title once this is ready to merge @maxcao13

k8s-ci-robot · 2025-02-24T18:19:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: maxcao13
Once this PR has been reviewed and has the lgtm label, please assign jbartosik for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

vertical-pod-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

We might want to add a few more that are combined disruption counters, e.g. in-place + eviction totals, but for now just add some separate counters to keep track of what in-place updates are doing.

For now, this just updates the mock with the new functions I added to the eviction interface. We need some in-place test cases.

TODO(jkyros): come back here and look at this after you get it working

So far this is just: - Make sure it scales when it can But we still need a bunch of other ones like - Test fallback to eviction - Test timeout/eviction when it gets stuck, etc

In the event that we can't perform the whole update, this calculates a set of updates that should be disruptionless and only queues that partial set, omitting the parts that would cause disruption.

Signed-off-by: Max Cao <[email protected]>

Because of kubernetes#7813, this commit reverts a lot of the changes that introducted logic that involved actuating in-place updates based on the containerResizePolicy. Signed-off-by: Max Cao <[email protected]>

Signed-off-by: Max Cao <[email protected]>

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Jan 7, 2025

k8s-ci-robot requested review from jbartosik and kwiesmueller January 7, 2025 18:10

k8s-ci-robot added area/vertical-pod-autoscaler needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 7, 2025

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 7, 2025

omerap12 reviewed Jan 10, 2025

View reviewed changes

maxcao13 force-pushed the maxcao13-inplace branch from 6ea7337 to 53d4006 Compare January 15, 2025 02:23

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 15, 2025

omerap12 reviewed Jan 15, 2025

View reviewed changes

vertical-pod-autoscaler/pkg/updater/eviction/pods_eviction_restriction.go Outdated Show resolved Hide resolved

maxcao13 mentioned this pull request Jan 17, 2025

Questions on AEP-4016: Support for in place updates in VPA #7722

Closed

adrianmoisey mentioned this pull request Jan 19, 2025

VPA: Implement in-place updates support #6652

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 20, 2025

jkyros and others added 5 commits February 3, 2025 12:53

VPA: Add UpdateModeInPlaceOrRecreate to types

8a59ac9

This just addes the UpdateModeInPlaceOrRecreate mode to the types so we can use it. I did not add InPlaceOnly, as that seemed contentious and it didn't seem like we had a good use case for it yet.

VPA: Allow admission-controller to validate in-place spec

cdfb9cb

Signed-off-by: Max Cao <[email protected]>

maxcao13 force-pushed the maxcao13-inplace branch from 2fa0144 to e739fd9 Compare February 3, 2025 20:54

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 3, 2025

raywainman mentioned this pull request Feb 7, 2025

Re-write parts of AEP-4016 (VPA with In-Place Updates) to align with the new In-Place Updates KEP changes #7813

Merged

sftim reviewed Feb 11, 2025

View reviewed changes

k8s-ci-robot changed the title ~~VPA: Implement in-place updates support~~ [WIP] VPA: Implement in-place updates support Feb 11, 2025

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 11, 2025

maxcao13 force-pushed the maxcao13-inplace branch from e739fd9 to b66e87d Compare February 24, 2025 18:19

jkyros added 4 commits February 24, 2025 10:38

VPA: Add metrics gauges for in-place updates

ed7759f

We might want to add a few more that are combined disruption counters, e.g. in-place + eviction totals, but for now just add some separate counters to keep track of what in-place updates are doing.

VPA: Update mocks to accommodate in-place VPA changes

8ad64fe

For now, this just updates the mock with the new functions I added to the eviction interface. We need some in-place test cases.

VPA: hack unit tests to account for in-place

3e9385f

TODO(jkyros): come back here and look at this after you get it working

VPA: Add e2e tests for in-place scaling

0f29cde

So far this is just: - Make sure it scales when it can But we still need a bunch of other ones like - Test fallback to eviction - Test timeout/eviction when it gets stuck, etc

maxcao13 force-pushed the maxcao13-inplace branch 3 times, most recently from 0ae50f0 to 753d0a4 Compare February 25, 2025 03:05

jkyros and others added 8 commits February 25, 2025 15:57

VPA: allow rule-breaking updates if disruptionless

ddf99f4

In the event that we can't perform the whole update, this calculates a set of updates that should be disruptionless and only queues that partial set, omitting the parts that would cause disruption.

VPA: only allow in-place if explicitly set

93b644a

VPA: Allow VPA updater to actuate recommendations in-place

e58ce3e

Signed-off-by: Max Cao <[email protected]>

VPA: Enable InPlacePodVerticalScaling feature flag on e2e

7e0a312

Signed-off-by: Max Cao <[email protected]>

VPA: Add in-place actuation and admission-controller e2e tests

d2c0c9a

Signed-off-by: Max Cao <[email protected]>

VPA: fix logs and cleanup TODOs according to review

57c2efa

Signed-off-by: Max Cao <[email protected]>

VPA: Revert changes that related to disruption/disruptionless changes

15c01c0

Because of kubernetes#7813, this commit reverts a lot of the changes that introducted logic that involved actuating in-place updates based on the containerResizePolicy. Signed-off-by: Max Cao <[email protected]>

VPA: Fix in-place actuation tests to align with updated AEP

50ba1c8

Signed-off-by: Max Cao <[email protected]>

maxcao13 force-pushed the maxcao13-inplace branch from 753d0a4 to 50ba1c8 Compare February 26, 2025 00:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] VPA: Implement in-place updates support #7673

[WIP] VPA: Implement in-place updates support #7673

maxcao13 commented Jan 7, 2025 •

edited

Loading

k8s-ci-robot commented Jan 7, 2025

k8s-triage-robot commented Jan 7, 2025

adrianmoisey commented Jan 7, 2025

omerap12 left a comment

maxcao13 commented Jan 15, 2025

omerap12 commented Jan 15, 2025

maxcao13 commented Jan 16, 2025

maxcao13 commented Jan 30, 2025 •

edited

Loading

sftim Feb 11, 2025

maxcao13 Feb 21, 2025 •

edited

Loading

sftim left a comment

sftim commented Feb 11, 2025

k8s-ci-robot commented Feb 24, 2025

[WIP] VPA: Implement in-place updates support #7673

Are you sure you want to change the base?

[WIP] VPA: Implement in-place updates support #7673

Conversation

maxcao13 commented Jan 7, 2025 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Jan 7, 2025

k8s-triage-robot commented Jan 7, 2025

adrianmoisey commented Jan 7, 2025

omerap12 left a comment

Choose a reason for hiding this comment

maxcao13 commented Jan 15, 2025

omerap12 commented Jan 15, 2025

maxcao13 commented Jan 16, 2025

maxcao13 commented Jan 30, 2025 • edited Loading

sftim Feb 11, 2025

Choose a reason for hiding this comment

maxcao13 Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

sftim left a comment

Choose a reason for hiding this comment

sftim commented Feb 11, 2025

k8s-ci-robot commented Feb 24, 2025

maxcao13 commented Jan 7, 2025 •

edited

Loading

maxcao13 commented Jan 30, 2025 •

edited

Loading

maxcao13 Feb 21, 2025 •

edited

Loading