the snapshot with error status cannot be removed successfully from cinder backend

[freesky-edward]

What happened:
when failed to create volume snapshot(error status cause by some reason from backend), I attempted to delete that volume-snapshot, the data in Kubernetes can be deleted successfully, however, the data in Cinder was not deleted successfully.

What you expected to happen:
I expect the snapshot data in backend should also be deleted when deleting a snapshot with error status. so that there is no residual useless data.

How to reproduce it (as minimally and precisely as possible):

1. create snapshot "kc create -f ../snapshot.yaml"
2. check snapshot status:
   root@kube-karbor:/opt/kube/kubernetes# kc describe volumesnapshot
   Name: snapshot-demo
   Namespace: default
   Labels:
   Annotations:
   API Version: volume-snapshot-data.external-storage.k8s.io/v1
   Kind: VolumeSnapshot
   Metadata:
   Cluster Name:
   Creation Timestamp: 2017-08-23T07:38:10Z
   Generation: 0
   Resource Version: 16203
   Self Link: /apis/volume-snapshot-data.external-storage.k8s.io/v1/namespaces/default/volumesnapshots/snapshot-demo
   UID: 0322489e-87d6-11e7-a4d5-fa163e1a1ced
   Spec:
   Persistent Volume Claim Name: cinder-claim1
   Snapshot Data Name: k8s-volume-snapshot-03601c96-87d6-11e7-a4c8-fa163e1a1ced
   Status:
   Conditions:
   Last Transition Time: 2017-08-23T07:38:10Z
   Message: Snapshot created succsessfully
   Reason:
   Status: True
   Type: Ready
   Creation Timestamp:
   Events:
   `
3. check cinder "openstack volume snapshot list"
   +--------------------------------------+-------------------------------------------------------------+---------------------+--------+------+
   | ID | Name | Description | Status | Size |
   +--------------------------------------+-------------------------------------------------------------+---------------------+--------+------+
   | 904bb9d1-00ea-4004-9899-a1c111b7a970 | pvc-1f93e356-87b7-11e7-a4d5-fa163e1a1ced1503473890359873214 | kubernetes snapshot | error | 1 |
   +--------------------------------------+-------------------------------------------------------------+---------------------+--------+------+
4. delete the snapshot "kc delete volumesnapshot snapshot-demo"
5. check snapshot in Kubernetes "kc get volumesnapshot"
   No resources found.
6. check snapshot in Cinder "openstack volume snapshot list"
   +--------------------------------------+-------------------------------------------------------------+---------------------+--------+------+
   | ID | Name | Description | Status | Size |
   +--------------------------------------+-------------------------------------------------------------+---------------------+--------+------+
   | 904bb9d1-00ea-4004-9899-a1c111b7a970 | pvc-1f93e356-87b7-11e7-a4d5-fa163e1a1ced1503473890359873214 | kubernetes snapshot | error | 1 |
   +--------------------------------------+-------------------------------------------------------------+---------------------+--------+------+

Anything else we need to know?:
logs when creating snapshot:
E0823 07:44:33.472653 15606 snapshotter.go:381] Failed to schedule the operation "createdefault/snapshot-democinder-claim1": Failed to create operation with name "createdefault/snapshot-democinder-claim1". An operation with that name failed at 2017-08-23 07:44:18.310211022 +0000 UTC m=+592.603224177. No retries permitted until 2017-08-23 07:46:18.310211022 +0000 UTC m=+712.603224177 (2m0s). Last error: "snapshot is not completed yet: current snapshot status is: error".
E0823 07:44:33.572948 15606 snapshotter.go:381] Failed to schedule the operation "createdefault/snapshot-democinder-claim1": Failed to create operation with name "createdefault/snapshot-democinder-claim1". An operation with that name failed at 2017-08-23 07:44:18.310211022 +0000 UTC m=+592.603224177. No retries permitted until 2017-08-23 07:46:18.310211022 +0000 UTC m=+712.603224177 (2m0s). Last error: "snapshot is not completed yet: current snapshot status is: error".

logs when deleting snapshot:
I0823 07:49:53.722128 15606 snapshot-controller.go:240] [CONTROLLER] OnDelete /apis/volume-snapshot-data.external-storage.k8s.io/v1/namespaces/default/volumesnapshots/snapshot-demo, snapshot name: default/snapshot-demo
I0823 07:49:53.722300 15606 desired_state_of_world.go:83] Deleting snapshot from desired state of world: default/snapshot-demo

Environment:

• Kubernetes version (use kubectl version): build from master
• Cloud provider or hardware configuration**: openstack
• OS (e.g. from /etc/os-release): ubuntu 16.04
• Kernel (e.g. uname -a): Linux kube-karbor 4.4.0-81-generic #104-Ubuntu SMP Wed Jun 14 08:17:06 UTC 2017 x86_64 x86_64 x86_64
• Install tools: N/A
• snapshot: build from master
• openstack: build from master

[rootfs]

This is a very interesting test case. It points to a case not addressed in snapshot controller.

During create:

• kubectl create snapshot triggers a VolumeSnapshot object created in dsw
• reconciler sees a new VolumeSnapshot object in dsw but missing in asw, it then calls snapshotter controllers to create it a VolumeSnapshotData, which represents a snapshot in the backend storage
• When backend successfully creates the snapshot, the VolumeSnapshotData is bound to the VolumeSnapshot. asw also records this object.

During delete:

• dsw deletes the snapshot object
• reconcilers sees a snapshot in asw but not in dsw, it calls snapshotter to delete the snapshot in the backend storage

Now in this test case, snapshot was not created, hence it was not added to asw. As a result, reconciler didn't call snapshotter to delete the snapshot in the backend storage.

@xing-yang has made changes to address some error paths. We'll fix this.

@freesky-edward thanks for providing the info

[xing-yang]

Thanks @freesky-edward for reporting this bug. I'm working on improving the error handling. Will take this test case into account.

[freesky-edward]

@rootfs
yeah, that's the cause , the volume data was not found in asw, so there was no request from reconciler. actually, Cinder can delete that data by command.
@xing-yang thanks.