cephfs: network fence does not work for cephfs

[Rakshith-R]

Describe the bug

network fence for cephfs does not work

rakshithr4:kubernetes-csi-addons$ k get networkfences.csiaddons.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: csiaddons.openshift.io/v1alpha1
  kind: NetworkFence
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"csiaddons.openshift.io/v1alpha1","kind":"NetworkFence","metadata":{"annotations":{},"name":"network-fence-sample"},"spec":{"cidrs":["10.90.89.66/32","11.67.12.42/32"],"driver":"rook-ceph.cephfs.csi.ceph.com","fenceState":"Unfenced","parameters":{"clusterID":"rook-ceph"},"secret":{"name":"rook-csi-cephfs-provisioner","namespace":"rook-ceph"}}}
    creationTimestamp: "2023-12-12T06:26:08Z"
    finalizers:
    - csiaddons.openshift.io/network-fence
    generation: 1
    name: network-fence-sample
    resourceVersion: "24785"
    uid: 0660c36d-f2cc-444a-a058-785344eb8f0f
  spec:
    cidrs:
    - 10.90.89.66/32
    - 11.67.12.42/32
    driver: rook-ceph.cephfs.csi.ceph.com
    fenceState: Unfenced
    parameters:
      clusterID: rook-ceph
    secret:
      name: rook-csi-cephfs-provisioner
      namespace: rook-ceph
  status:
    message: 'rpc error: code = InvalidArgument desc = missing ID field ''userID''
      in secrets'
    result: Failed
kind: List
metadata:
  resourceVersion: "

Environment details

• Image/version of Ceph CSI driver : 3.10.0
• Helm chart version : -
• Kernel version : -
• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
  krbd or rbd-nbd) : -
• Kubernetes cluster version : -
• Ceph cluster version : -

Steps to reproduce

Steps to reproduce the behavior:

1. Setup details: setup csiaddons and rook-ceph with latest cephcsi 3.10.0
2. Create network fence CR for cephfs
3. See error

Actual results

The container expects UserID while cephfs secrets contain AdminID.

Expected behavior

NetworkFence is successful

Logs

2023-12-12T06:26:13.727Z        ERROR   failed to fence cluster network {"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"network-fence-sample"}, "namespace": "", "name": "network-fence-sample", "reconcileID": "de23838d-d52d-48de-9099-afbf5d75a52f", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "CIDRs": ["10.90.89.66/32", "11.67.12.42/32"], "error": "rpc error: code = InvalidArgument desc = missing ID field 'userID' in secrets"}
github.com/csi-addons/kubernetes-csi-addons/controllers/csiaddons.(*NetworkFenceReconciler).Reconcile
        /workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/controllers/csiaddons/networkfence_controller.go:148
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226

I1212 06:28:52.526483       1 utils.go:164] ID: 43 GRPC call: /fence.FenceController/UnfenceClusterNetwork
I1212 06:28:52.526732       1 utils.go:165] ID: 43 GRPC request: {"cidrs":[{"cidr":"10.90.89.66/32"},{"cidr":"11.67.12.42/32"}],"parameters":{"clusterID":"rook-ceph"},"secrets":"***stripped***"}
E1212 06:28:52.526762       1 utils.go:169] ID: 43 GRPC error: rpc error: code = InvalidArgument desc = missing ID field 'userID' in secrets

cc @nixpanic @riya-singhal31

[Rakshith-R]

Please add logs of csi-addons controller csi-addons sidecar and cephcsi container.

Did you execute the operation with a live cephfs PVC mounted ?
Try to read write after executing the operation ?

[riya-singhal31]

@Rakshith-R PTAL, these are the logs from csi-addons container, logs shows successful operation

2023-12-17T19:47:04.039Z	INFO	Adding finalizer	{"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"csi-rbdplugin-provisioner-58c477b559-vvrch","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "csi-rbdplugin-provisioner-58c477b559-vvrch", "reconcileID": "4f75278c-bc1b-4656-b8fc-7ad7cc0c44cb", "NodeID": "ip-10-0-38-99.ec2.internal", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.129.2.30:9070"}
2023-12-17T19:47:04.046Z	INFO	Connecting to sidecar	{"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"csi-rbdplugin-provisioner-58c477b559-vvrch","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "csi-rbdplugin-provisioner-58c477b559-vvrch", "reconcileID": "4f75278c-bc1b-4656-b8fc-7ad7cc0c44cb", "NodeID": "ip-10-0-38-99.ec2.internal", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.129.2.30:9070"}
2023-12-17T19:47:04.048Z	INFO	Successfully connected to sidecar	{"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"csi-rbdplugin-provisioner-58c477b559-vvrch","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "csi-rbdplugin-provisioner-58c477b559-vvrch", "reconcileID": "4f75278c-bc1b-4656-b8fc-7ad7cc0c44cb", "NodeID": "ip-10-0-38-99.ec2.internal", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.129.2.30:9070"}
2023-12-17T19:47:04.048Z	INFO	Added connection to connection pool	{"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"csi-rbdplugin-provisioner-58c477b559-vvrch","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "csi-rbdplugin-provisioner-58c477b559-vvrch", "reconcileID": "4f75278c-bc1b-4656-b8fc-7ad7cc0c44cb", "NodeID": "ip-10-0-38-99.ec2.internal", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.129.2.30:9070"}
2023-12-17T20:02:14.404Z	INFO	FenceClusterNetwork Request	{"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"network-fence-sample-cephfs1"}, "namespace": "", "name": "network-fence-sample-cephfs1", "reconcileID": "f57fc16e-4454-47a4-9ad0-d40578d95b93", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "CIDRs": ["10.130.2.28/32"], "namespaced name": "/network-fence-sample-cephfs1"}
2023-12-17T20:02:14.404Z	INFO	adding finalizer to NetworkFence object	{"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"network-fence-sample-cephfs1"}, "namespace": "", "name": "network-fence-sample-cephfs1", "reconcileID": "f57fc16e-4454-47a4-9ad0-d40578d95b93", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "CIDRs": ["10.130.2.28/32"], "Finalizer": "csiaddons.openshift.io/network-fence"}
2023-12-17T20:02:15.432Z	INFO	FenceClusterNetwork Request Succeeded	{"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"network-fence-sample-cephfs1"}, "namespace": "", "name": "network-fence-sample-cephfs1", "reconcileID": "f57fc16e-4454-47a4-9ad0-d40578d95b93", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "CIDRs": ["10.130.2.28/32"]}

riyasinghal@rsinghal-mac examples % kubectl get networkfence.csiaddons.openshift.io                                
NAME                           DRIVER                          CIDRS                FENCESTATE   AGE     RESULT
network-fence-sample-cephfs1   rook-ceph.cephfs.csi.ceph.com   ["10.130.2.28/32"]   Fenced       3m49s   Succeeded

cc: @nixpanic

[Rakshith-R]

Please add logs of csi-addons controller csi-addons sidecar and cephcsi container.

Did you execute the operation with a live cephfs PVC mounted ? Try to read write after executing the operation ?

Where are the logs for other two containers?
You didn't answer the remaining two questions.

[riya-singhal31]

Please add logs of csi-addons controller csi-addons sidecar and cephcsi container.
Did you execute the operation with a live cephfs PVC mounted ? Try to read write after executing the operation ?

Where are the logs for other two containers? You didn't answer the remaining two questions.

riyasinghal@rsinghal-mac examples % kubectl logs csi-cephfsplugin-provisioner-698c58c9bb-ftgsm -c csi-addons -n rook-ceph
I1218 08:58:52.482723       1 client.go:80] Probing CSI driver for readiness
I1218 08:58:52.500105       1 server.go:90] Listening for CSI-Addons requests on address: [::]:9070
I1218 08:58:52.500297       1 leaderelection.go:250] attempting to acquire leader lease rook-ceph/rook-ceph-cephfs-csi-ceph-com-csi-addons...

iyasinghal@rsinghal-mac examples % kubectl logs csi-cephfsplugin-provisioner-698c58c9bb-ftgsm -c csi-cephfsplugin -n rook-ceph
E1218 09:01:37.885542       1 omap.go:79] ID: 22 Req-ID: pvc-a39bf200-0683-49f6-9b2e-6aa991ac8d23 omap not found (pool="myfs-metadata", namespace="csi", name="csi.volumes.default"): rados: ret=-2, No such file or directory

riyasinghal@rsinghal-mac examples % kubectl get pvc,pods                                                                 
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cephfs-pvc   Bound    pvc-a39bf200-0683-49f6-9b2e-6aa991ac8d23   1Gi        RWO            rook-cephfs    2m2s

NAME                     READY   STATUS    RESTARTS   AGE
pod/csicephfs-demo-pod   1/1     Running   0          116s

yes, applied the operations with live pvc.

[Rakshith-R]

https://github.com/ceph/ceph-csi/releases/tag/v3.10.1
this should be probably fixed with v3.10.1 release