rbd: fix encrypted PVC with metadata KMS cannot be deleted #5149

zerotens · 2025-02-16T13:32:21Z

Describe what this PR does

Creating an encrypted Persistent Volume Claim with reclaim policy Retain, the Persistent Volume cannot be deleted after the Namespace with the corresponding secret of the encrypted volume was deleted.

CSI DeleteVolume Request -> calls GenVolFromVolID -> calls generateVolumeFromVolumeID -> calls rbdVol.configureBlockEncryption which initiates the metadata kms provider

This fix moves the logic from the kms class initialization to the method where the encryption key is needed.
As the Delete Volume CSI request does not need or call the FetchDEK method, the volume get's deleted successfully.

Is there anything that requires special attention

Do you have any questions?

Is the change backward compatible?

Are there concerns around backward compatibility?

Related issues

Fixes: #5148

Checklist:

Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
Reviewed the developer guide on Submitting a Pull
Request
Pending release
notes
updated with breaking and/or notable changes for the next major release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

Rakshith-R

thanks for debugging and fixing the issue !

Just a small nit.

internal/kms/secretskms.go

Rakshith-R

Thanks

iPraveenParihar · 2025-02-20T06:23:40Z

/test ci/centos/mini-e2e/k8s-1.32

Madhu-1 · 2025-02-20T08:01:31Z

/test ci/centos/mini-e2e/k8s-1.32

@iPraveenParihar please provide details in a comment about the cause of failure as well, it helps to identify unknown bugs

nixpanic · 2025-02-20T10:23:28Z

@Mergifyio rebase

mergify · 2025-02-20T10:23:42Z

rebase

✅ Branch has been successfully rebased

nixpanic · 2025-02-20T10:23:52Z

@Mergifyio queue

mergify · 2025-02-20T10:23:59Z

queue

🛑 The pull request has been removed from the queue `default`

The merge conditions cannot be satisfied due to failing checks.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

ceph-csi-bot · 2025-02-20T10:24:15Z

/test ci/centos/k8s-e2e-external-storage/1.32

ceph-csi-bot · 2025-02-20T10:24:15Z

/test ci/centos/mini-e2e-helm/k8s-1.32

ceph-csi-bot · 2025-02-20T10:24:16Z

/test ci/centos/k8s-e2e-external-storage/1.30

ceph-csi-bot · 2025-02-20T10:24:16Z

/test ci/centos/mini-e2e/k8s-1.32

ceph-csi-bot · 2025-02-20T10:24:16Z

/test ci/centos/mini-e2e-helm/k8s-1.30

ceph-csi-bot · 2025-02-20T10:24:17Z

/test ci/centos/mini-e2e/k8s-1.30

ceph-csi-bot · 2025-02-20T10:24:17Z

/test ci/centos/k8s-e2e-external-storage/1.31

ceph-csi-bot · 2025-02-20T10:24:17Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2025-02-20T10:24:18Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2025-02-20T10:24:18Z

/test ci/centos/mini-e2e-helm/k8s-1.31

ceph-csi-bot · 2025-02-20T10:24:19Z

/test ci/centos/mini-e2e/k8s-1.31

mergify · 2025-02-20T12:17:20Z

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

❌ ci/centos/mini-e2e/k8s-1.32

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue

nixpanic · 2025-02-20T12:51:23Z

/retest ci/centos/mini-e2e/k8s-1.32

nixpanic · 2025-02-20T12:53:09Z

/retest ci/centos/mini-e2e/k8s-1.32

The logs of the failure do not immediate show what the issue is, I think:

  �[1mSTEP:�[0m create a PVC and check PVC/PV metadata on RBD image �[38;5;243m@ 02/20/25 11:57:08.555�[0m
  I0220 11:57:08.558944 86817 pvc.go:60] Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc  rbd-1827    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[group:test] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state
  I0220 11:57:08.558987 86817 pvc.go:63] waiting for PVC rbd-pvc (0 seconds elapsed)
  I0220 11:57:08.563534 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563572 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563799 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563817 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a
  I0220 11:57:08.563825 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563833 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563840 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-ca574bf2-9632-49b9-8125-66951fc004c2
  I0220 11:57:10.559694 86817 pvc.go:63] waiting for PVC rbd-pvc (2 seconds elapsed)
  I0220 11:57:10.563681 86817 pv.go:468] Waiting for PV pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to bind to PVC rbd-pvc
  I0220 11:57:10.563735 86817 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
  I0220 11:57:10.567465 86817 pv.go:801] PersistentVolumeClaim rbd-pvc found and phase=Bound (3.716004ms)
  I0220 11:57:10.567495 86817 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to have phase Bound
  I0220 11:57:10.569559 86817 pv.go:771] PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb found and phase=Bound (2.052301ms)
  I0220 11:57:10.574121 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.574146 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.574201 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.685176 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. |  grep -c ^csi.volume.] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.685215 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.685248 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool+%7C+grep+-v+default+%7C+grep+-v+csi.volume.group.+%7C++grep+-c+%5Ecsi.volume.&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.807774 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.807838 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.807873 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.917795 86817 utils.go:251] additional debug info: rados ls command output: rbd_header.13de5595a987
  rbd_directory
  rbd_info
  csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5
  rbd_task
  csi.volumes.default
  rbd_trash
  csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1
  rbd_id.csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5
  , stdErr: 
  �[38;5;9m[FAILED]�[0m in [It] - /go/src/github.com/ceph/ceph-csi/e2e/utils.go:254 �[38;5;243m@ 02/20/25 11:57:10.918�[0m

nixpanic · 2025-02-20T12:54:23Z

@Mergifyio requeue

mergify · 2025-02-20T12:54:31Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Madhu-1 · 2025-02-20T13:25:14Z

/retest ci/centos/mini-e2e/k8s-1.32

The logs of the failure do not immediate show what the issue is, I think:

  �[1mSTEP:�[0m create a PVC and check PVC/PV metadata on RBD image �[38;5;243m@ 02/20/25 11:57:08.555�[0m
  I0220 11:57:08.558944 86817 pvc.go:60] Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc  rbd-1827    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[group:test] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state
  I0220 11:57:08.558987 86817 pvc.go:63] waiting for PVC rbd-pvc (0 seconds elapsed)
  I0220 11:57:08.563534 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563572 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563799 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563817 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a
  I0220 11:57:08.563825 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563833 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563840 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-ca574bf2-9632-49b9-8125-66951fc004c2
  I0220 11:57:10.559694 86817 pvc.go:63] waiting for PVC rbd-pvc (2 seconds elapsed)
  I0220 11:57:10.563681 86817 pv.go:468] Waiting for PV pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to bind to PVC rbd-pvc
  I0220 11:57:10.563735 86817 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
  I0220 11:57:10.567465 86817 pv.go:801] PersistentVolumeClaim rbd-pvc found and phase=Bound (3.716004ms)
  I0220 11:57:10.567495 86817 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to have phase Bound
  I0220 11:57:10.569559 86817 pv.go:771] PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb found and phase=Bound (2.052301ms)
  I0220 11:57:10.574121 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.574146 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.574201 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.685176 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. |  grep -c ^csi.volume.] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.685215 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.685248 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool+%7C+grep+-v+default+%7C+grep+-v+csi.volume.group.+%7C++grep+-c+%5Ecsi.volume.&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.807774 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.807838 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.807873 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.917795 86817 utils.go:251] additional debug info: rados ls command output: rbd_header.13de5595a987
  rbd_directory
  rbd_info
  csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5
  rbd_task
  csi.volumes.default
  rbd_trash
  csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1
  rbd_id.csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5
  , stdErr: 
  �[38;5;9m[FAILED]�[0m in [It] - /go/src/github.com/ceph/ceph-csi/e2e/utils.go:254 �[38;5;243m@ 02/20/25 11:57:10.918�[0m

[38;5;9m[FAILED] failed to validate omap count for rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. | grep -c ^csi.volume.: expected omap object count 1, got 2�[0m

i think there is a omap leak.

Rakshith-R · 2025-02-25T09:54:20Z

/retest ci/centos/mini-e2e/k8s-1.32

The logs of the failure do not immediate show what the issue is, I think:

  �[1mSTEP:�[0m create a PVC and check PVC/PV metadata on RBD image �[38;5;243m@ 02/20/25 11:57:08.555�[0m
  I0220 11:57:08.558944 86817 pvc.go:60] Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc  rbd-1827    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[group:test] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state
  I0220 11:57:08.558987 86817 pvc.go:63] waiting for PVC rbd-pvc (0 seconds elapsed)
  I0220 11:57:08.563534 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563572 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563799 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563817 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a
  I0220 11:57:08.563825 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563833 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563840 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-ca574bf2-9632-49b9-8125-66951fc004c2
  I0220 11:57:10.559694 86817 pvc.go:63] waiting for PVC rbd-pvc (2 seconds elapsed)
  I0220 11:57:10.563681 86817 pv.go:468] Waiting for PV pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to bind to PVC rbd-pvc
  I0220 11:57:10.563735 86817 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
  I0220 11:57:10.567465 86817 pv.go:801] PersistentVolumeClaim rbd-pvc found and phase=Bound (3.716004ms)
  I0220 11:57:10.567495 86817 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to have phase Bound
  I0220 11:57:10.569559 86817 pv.go:771] PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb found and phase=Bound (2.052301ms)
  I0220 11:57:10.574121 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.574146 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.574201 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.685176 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. |  grep -c ^csi.volume.] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.685215 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.685248 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool+%7C+grep+-v+default+%7C+grep+-v+csi.volume.group.+%7C++grep+-c+%5Ecsi.volume.&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.807774 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.807838 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.807873 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.917795 86817 utils.go:251] additional debug info: rados ls command output: rbd_header.13de5595a987
  rbd_directory
  rbd_info
  csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5
  rbd_task
  csi.volumes.default
  rbd_trash
  csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1
  rbd_id.csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5
  , stdErr: 
  �[38;5;9m[FAILED]�[0m in [It] - /go/src/github.com/ceph/ceph-csi/e2e/utils.go:254 �[38;5;243m@ 02/20/25 11:57:10.918�[0m

[38;5;9m[FAILED] failed to validate omap count for rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. | grep -c ^csi.volume.: expected omap object count 1, got 2�[0m

i think there is a omap leak.

I don't think this issue related to this pr.
It failed before RBD tests related to encryption were encountere.

  �[1mSTEP:�[0m create a PVC and check PVC/PV metadata on RBD image �[38;5;243m@ 02/20/25 11:57:08.555�[0m
  I0220 11:57:08.558944 86817 pvc.go:60] Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc  rbd-1827    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[group:test] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state
  I0220 11:57:08.558987 86817 pvc.go:63] waiting for PVC rbd-pvc (0 seconds elapsed)
  I0220 11:57:08.563534 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563572 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563799 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563817 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-e4aa7cac-30f1-4f5e-bc5c-af17b8764c9a
  I0220 11:57:08.563825 86817 pvc.go:85] PVC rbd-pvc Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  I0220 11:57:08.563833 86817 pvc.go:85] PVC rbd-pvc Event: Provisioning - External provisioner is provisioning volume for claim "rbd-1827/rbd-pvc"
  I0220 11:57:08.563840 86817 pvc.go:85] PVC rbd-pvc Event: ProvisioningSucceeded - Successfully provisioned volume pvc-ca574bf2-9632-49b9-8125-66951fc004c2
  I0220 11:57:10.559694 86817 pvc.go:63] waiting for PVC rbd-pvc (2 seconds elapsed)
  I0220 11:57:10.563681 86817 pv.go:468] Waiting for PV pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to bind to PVC rbd-pvc
  I0220 11:57:10.563735 86817 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
  I0220 11:57:10.567465 86817 pv.go:801] PersistentVolumeClaim rbd-pvc found and phase=Bound (3.716004ms)
  I0220 11:57:10.567495 86817 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb to have phase Bound
  I0220 11:57:10.569559 86817 pv.go:771] PersistentVolume pvc-8a23f93d-d21b-4d11-9485-c7f048ea55fb found and phase=Bound (2.052301ms)
  I0220 11:57:10.574121 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.574146 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.574201 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rbd+ls+--format%3Djson+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.685176 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool | grep -v default | grep -v csi.volume.group. |  grep -c ^csi.volume.] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.685215 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.685248 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool+%7C+grep+-v+default+%7C+grep+-v+csi.volume.group.+%7C++grep+-c+%5Ecsi.volume.&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.807774 86817 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c rados ls --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-56ff7b659c-lvrcs ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
  I0220 11:57:10.807838 86817 exec_util.go:64] ExecWithOptions: Clientset creation
  I0220 11:57:10.807873 86817 exec_util.go:80] ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-56ff7b659c-lvrcs/exec?command=%2Fbin%2Fsh&command=-c&command=rados+ls+--pool%3Dreplicapool&container=rook-ceph-tools&stderr=true&stdout=true)
  I0220 11:57:10.917795 86817 utils.go:251] additional debug info: rados ls command output: rbd_header.13de5595a987
  rbd_directory
  rbd_info
  csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5
  rbd_task
  csi.volumes.default
  rbd_trash
  csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1
  rbd_id.csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5
  , stdErr: 
  �[38;5;9m[FAILED]�[0m in [It] - /go/src/github.com/ceph/ceph-csi/e2e/utils.go:254 �[38;5;243m@ 02/20/25 11:57:10.918�[0m

iPraveenParihar · 2025-02-25T09:55:16Z

I0220 11:57:10.917795 86817 utils.go:251] additional debug info: rados ls command output: rbd_header.13de5595a987
rbd_directory
rbd_info
csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5
rbd_task
csi.volumes.default
rbd_trash
csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1
rbd_id.csi-vol-aec40a24-45ed-4919-97ca-b2c5b00517b5

csi.volume.aec40a24-45ed-4919-97ca-b2c5b00517b5 is from failed test. The stale entry csi.volume.860bd5be-e492-4bd0-b3de-7ce56fa535e1 is not present in collected logs.

Rakshith-R · 2025-02-25T10:06:25Z

/retest ci/centos/mini-e2e/k8s-1.32

Signed-off-by: Zerotens <[email protected]>

ceph-csi-bot · 2025-02-25T11:57:12Z

/test ci/centos/k8s-e2e-external-storage/1.30

ceph-csi-bot · 2025-02-25T11:57:13Z

/test ci/centos/mini-e2e-helm/k8s-1.30

ceph-csi-bot · 2025-02-25T11:57:13Z

/test ci/centos/mini-e2e/k8s-1.30

ceph-csi-bot · 2025-02-25T11:57:14Z

/test ci/centos/k8s-e2e-external-storage/1.31

ceph-csi-bot · 2025-02-25T11:57:14Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2025-02-25T11:57:14Z

/test ci/centos/k8s-e2e-external-storage/1.32

ceph-csi-bot · 2025-02-25T11:57:14Z

/test ci/centos/mini-e2e-helm/k8s-1.31

ceph-csi-bot · 2025-02-25T11:57:15Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2025-02-25T11:57:15Z

/test ci/centos/mini-e2e-helm/k8s-1.32

ceph-csi-bot · 2025-02-25T11:57:15Z

/test ci/centos/mini-e2e/k8s-1.31

ceph-csi-bot · 2025-02-25T11:57:15Z

/test ci/centos/mini-e2e/k8s-1.32

mergify bot added component/rbd Issues related to RBD bug Something isn't working labels Feb 16, 2025

zerotens force-pushed the bugfix/secretmetadatakms-volumedelete branch 3 times, most recently from ec64661 to 7e59b8f Compare February 16, 2025 14:33

Rakshith-R requested changes Feb 17, 2025

View reviewed changes

internal/kms/secretskms.go Outdated Show resolved Hide resolved

nixpanic added the backport-to-release-v3.13 Label to backport from devel to release-v3.13 branch label Feb 17, 2025

zerotens force-pushed the bugfix/secretmetadatakms-volumedelete branch from 7e59b8f to 5839c6f Compare February 17, 2025 17:39

zerotens requested a review from Rakshith-R February 17, 2025 17:42

Rakshith-R approved these changes Feb 19, 2025

View reviewed changes

Rakshith-R requested a review from a team February 19, 2025 09:24

iPraveenParihar approved these changes Feb 20, 2025

View reviewed changes

nixpanic force-pushed the bugfix/secretmetadatakms-volumedelete branch from 5839c6f to a8c53a1 Compare February 20, 2025 10:23

mergify bot added the ok-to-test Label to trigger E2E tests label Feb 20, 2025

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Feb 20, 2025

Rakshith-R mentioned this pull request Feb 25, 2025

rbd: omap leak caught in jenkins ci #5175

Open

rbd: fix encrypted PVC with metadata KMS cannot be deleted

733ba3c

Signed-off-by: Zerotens <[email protected]>

zerotens force-pushed the bugfix/secretmetadatakms-volumedelete branch from a8c53a1 to 733ba3c Compare February 25, 2025 11:56

mergify bot added the ok-to-test Label to trigger E2E tests label Feb 25, 2025

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Feb 25, 2025

mergify bot merged commit 5b587c9 into ceph:devel Feb 25, 2025
37 checks passed

mergify bot mentioned this pull request Feb 25, 2025

rbd: fix encrypted PVC with metadata KMS cannot be deleted (backport #5149) #5176

Merged

6 tasks

rbd: fix encrypted PVC with metadata KMS cannot be deleted #5149

rbd: fix encrypted PVC with metadata KMS cannot be deleted #5149

Conversation

zerotens commented Feb 16, 2025

Describe what this PR does

Is there anything that requires special attention

Related issues

Rakshith-R left a comment

Choose a reason for hiding this comment

Rakshith-R left a comment

Choose a reason for hiding this comment

iPraveenParihar commented Feb 20, 2025

Madhu-1 commented Feb 20, 2025

nixpanic commented Feb 20, 2025

mergify bot commented Feb 20, 2025

✅ Branch has been successfully rebased

nixpanic commented Feb 20, 2025

mergify bot commented Feb 20, 2025 • edited Loading

🛑 The pull request has been removed from the queue default

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

ceph-csi-bot commented Feb 20, 2025

mergify bot commented Feb 20, 2025

nixpanic commented Feb 20, 2025

nixpanic commented Feb 20, 2025

nixpanic commented Feb 20, 2025

mergify bot commented Feb 20, 2025

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Madhu-1 commented Feb 20, 2025

Rakshith-R commented Feb 25, 2025

iPraveenParihar commented Feb 25, 2025

Rakshith-R commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

ceph-csi-bot commented Feb 25, 2025

mergify bot commented Feb 20, 2025 •

edited

Loading

🛑 The pull request has been removed from the queue `default`