csi-rbdplugin "supported_features: no such file or directory" error. #2678

humpalu · 2021-11-23T15:22:47Z

Describe the bug

I get the following error message from the "csi-rbdplugin" container in all "ceph-csi-rbd-nodeplugin" pods.

I1123 14:51:44.550509   12355 mount_linux.go:207] Cannot run systemd-run, assuming non-systemd OS
I1123 14:51:44.550539   12355 mount_linux.go:208] systemd-run output: System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to create bus connection: Host is down
, failed with: exit status 1
E1123 14:51:44.553355   12355 rbd_util.go:241] reading file "/sys/bus/rbd/supported_features" failed: open /sys/bus/rbd/supported_features: no such file or directory
F1123 14:51:44.553380   12355 driver.go:163] open /sys/bus/rbd/supported_features: no such file or directory
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc0001b6001, 0xc0001e8c80, 0x6d, 0x24d)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x31a8c00, 0xc000000003, 0x0, 0x0, 0xc000544cb0, 0x1, 0x26e3858, 0x9, 0xa3, 0x418e00)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:975 +0x1e5
k8s.io/klog/v2.(*loggingT).printDepth(0x31a8c00, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc0002cc560, 0x1, 0x1)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:735 +0x185
k8s.io/klog/v2.FatalDepth(...)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:1500
github.com/ceph/ceph-csi/internal/util/log.FatalLogMsg(0xc000048280, 0x3f, 0x0, 0x0, 0x0)
        /go/src/github.com/ceph/ceph-csi/internal/util/log/log.go:59 +0x118
github.com/ceph/ceph-csi/internal/rbd.(*Driver).Run(0xc0000ae990, 0x31a8d20)
        /go/src/github.com/ceph/ceph-csi/internal/rbd/driver.go:163 +0xda5
main.main()
        /go/src/github.com/ceph/ceph-csi/cmd/cephcsi.go:223 +0x3a5

Environment details

Image/version of Ceph CSI driver : Canary and Git version: b2099eb
Helm chart version : ceph-csi-rbd-3.4.0
Kernel version :
Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
krbd or rbd-nbd) : krbd
Kubernetes cluster version : v1.21.2
Ceph cluster version : 14

Steps to reproduce

Steps to reproduce the behavior:

helm install
helm install -f rbd_values.yaml --namespace "ceph-csi-rbd" "ceph-csi-rbd" ceph-csi/ceph-csi-rbd

rbd_values.yaml contains the following changes

csiConfig:
  - clusterID: "xxxxx-xxxxe-4758-8aea-8ccf52xxx9f"
    monitors:
      - "10.33.xx.xx:6789"
      - "10.33.xx.xx:6789"
      - "10.33.xx.xx:6789"

storageClass:
  # Specifies whether the storageclass should be created
  create: true
  name: csi-rbd-sc
  clusterID: xxxxx-xxxxe-4758-8aea-8ccf52xxx9f

pool: "vmpool"

secret:
  # Specifies whether the secret should be created
  create: true
  name: csi-rbd-secret
  userID: admin
  userKey: "XXXXXXX"

Actual results

ceph-csi-rbd-nodeplugin-mbcvk 2/3 CrashLoopBackOff 18 71m
ceph-csi-rbd-nodeplugin-md9v5 2/3 CrashLoopBackOff 18 71m

Expected behavior

I expect to all pods are running :-)

Logs

ceph-csi-rbd-nodeplugin/csi-rbdplugin

I1123 15:17:14.633032   20442 cephcsi.go:168] Driver version: canary and Git version: b2099eb3b1a46ed326ded1aa5b2601a418726d91
I1123 15:17:14.633307   20442 cephcsi.go:186] Initial PID limit is set to -1
I1123 15:17:14.633430   20442 cephcsi.go:192] Reconfigured PID limit to -1 (max)
I1123 15:17:14.633593   20442 cephcsi.go:217] Starting driver type: rbd with name: rbd.csi.ceph.com
I1123 15:17:14.650084   20442 mount_linux.go:207] Cannot run systemd-run, assuming non-systemd OS
I1123 15:17:14.650119   20442 mount_linux.go:208] systemd-run output: System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to create bus connection: Host is down
, failed with: exit status 1
E1123 15:17:14.653174   20442 rbd_util.go:241] reading file "/sys/bus/rbd/supported_features" failed: open /sys/bus/rbd/supported_features: no such file or directory
F1123 15:17:14.653208   20442 driver.go:163] open /sys/bus/rbd/supported_features: no such file or directory
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000010001, 0xc0000e4500, 0x6d, 0x24d)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x31a8c00, 0xc000000003, 0x0, 0x0, 0xc0000b78f0, 0x1, 0x26e3858, 0x9, 0xa3, 0x418e00)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:975 +0x1e5
k8s.io/klog/v2.(*loggingT).printDepth(0x31a8c00, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc0003685a0, 0x1, 0x1)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:735 +0x185
k8s.io/klog/v2.FatalDepth(...)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:1500
github.com/ceph/ceph-csi/internal/util/log.FatalLogMsg(0xc000048580, 0x3f, 0x0, 0x0, 0x0)
        /go/src/github.com/ceph/ceph-csi/internal/util/log/log.go:59 +0x118
github.com/ceph/ceph-csi/internal/rbd.(*Driver).Run(0xc0000b0b10, 0x31a8d20)
        /go/src/github.com/ceph/ceph-csi/internal/rbd/driver.go:163 +0xda5
main.main()
        /go/src/github.com/ceph/ceph-csi/cmd/cephcsi.go:223 +0x3a5

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x31a8c00)
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
        /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/klog/v2/klog.go:420 +0xdf

goroutine 37 [sleep]:
time.Sleep(0x6fc23ac00)
        /usr/local/go/src/runtime/time.go:193 +0xd2
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
        /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:63 +0x45
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
        /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:62 +0x35

The text was updated successfully, but these errors were encountered:

Madhu-1 · 2021-11-23T15:47:59Z

@pkalever PTAL

humpalu · 2021-11-23T18:22:42Z

If my idea is correct, the /sys/bus/rbd/supported_features file should be installed from package.
I checked , none of my worker node has that file.
So the file is installed from package or generated by ceph client?

nixpanic · 2021-11-24T07:41:47Z

The /sys/bus/rbd/supported_features files should get automatically created when the rbd kernel module is loaded. Can you confirm the module is loaded on the worker nodes? If it is not loaded yet, can you manually load it and verify that the file exists and the pod start again?

There might be an issue in the order or detecting the file, and loading the kernel module for some reason. We would need to understand why that happens.

humpalu · 2021-11-24T09:01:42Z

I can confirm, the kernel module is loaded on the host system

root@node-xxx:~# lsmod | grep rbd
rbd                    69632  0
libceph               245760  1 rbd
root@node-xxx:~# modinfo rbd
filename:       /lib/modules/4.4.0-210-generic/kernel/drivers/block/rbd.ko
license:        GPL
description:    RADOS Block Device (RBD) driver
author:         Jeff Garzik <[email protected]>
author:         Yehuda Sadeh <[email protected]>
author:         Sage Weil <[email protected]>
author:         Alex Elder <[email protected]>
srcversion:     037573996A1AC4BA9831E78
depends:        libceph
retpoline:      Y
intree:         Y
vermagic:       4.4.0-210-generic SMP mod_unload modversions
parm:           single_major:Use a single major number for all rbd devices (default: false) (bool)

root@node-xxx:~# ls -1 /sys/bus/rbd/
add
devices
drivers
drivers_autoprobe
drivers_probe
remove
uevent

nixpanic · 2021-11-24T16:15:00Z

Maybe the /sys/bus/rbd/supported_features was added in a more recent kernel version? You seem to be on Linux 4.4.0, which is relatively old.

@pkalever can you update the check for features with some sensible defaults for when the supported_features file is missing?

github-actions · 2021-12-24T21:06:23Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions · 2021-12-31T21:06:26Z

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

Madhu-1 · 2022-01-04T08:26:34Z

reopening @pkalever PTAL when you get some time.

Madhu-1 · 2022-01-04T08:27:28Z

this should be a blocker for the release as it's tagged for 3.5.0? @ceph/ceph-csi-maintainers

humblec · 2022-01-04T09:53:49Z

this should be a blocker for the release as it's tagged for 3.5.0? @ceph/ceph-csi-maintainers

Considering this kernel is in use, it looks like we have to consider this as a blocker.
@pkalever can you please take a look at this ? and may be a workaround ?

nixpanic · 2022-01-06T13:11:42Z

As this is also a bug in 3.4.0, we do not need to block releasing 3.5.0 for this. A bugfix can get backported to a minor release later.

Madhu-1 · 2022-01-07T05:10:09Z

@nixpanic this code is present only in devel branch https://github.com/ceph/ceph-csi/blob/devel/internal/rbd/rbd_util.go#L229-L254 not in 3.4

nixpanic · 2022-01-07T12:55:58Z

@nixpanic this code is present only in devel branch https://github.com/ceph/ceph-csi/blob/devel/internal/rbd/rbd_util.go#L229-L254 not in 3.4

Hmm, in that case it would be good to have a fix in 3.5.0, or some note about a minimal kernel version (for the rbd module).

This was reported against ceph-csi-rbd-3.4.0, so I assumed it is included in 3.4.0 as well 🤷‍♂️

github-actions · 2022-02-06T21:06:02Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

pkalever · 2022-02-09T07:02:39Z

@Madhu-1 @nixpanic

Sorry guys! looks like this went into some spam email box or somehow I ignored on this email thread notification may be based on the title. Anyways I will look into this now.

I see the /sys/bus/rbd/supported_features is part of linux kernel v4.11.0

commit 8767b293a4ab6632f9288f34bcf2ab9ba20dca3a
Author: Ilya Dryomov <[email protected]>
Date:   Thu Mar 2 19:56:57 2017 +0100

    rbd: supported_features bus attribute
    
    ... so that userspace can generate meaningful error messages and spell
    out unsupported features that need to be disabled.
    
    Signed-off-by: Ilya Dryomov <[email protected]>
    Reviewed-by: Sage Weil <[email protected]>

So, What route do we want to take here?

The difficulty part is that what should be the minimum sensible values?
RBD_FEATURE_LAYERING is added around linux v3.6, RBD_FEATURE_STRIPINGV2 is added around v3.9, RBD_FEATURE_EXCLUSIVE_LOCK is added around v4.8, and RBD_FEATURE_DATA_POOL around v4.10
Shoud we implement some intellegence in Ceph-CSI to check kernel version and decide the min suported feature list?
Or should we just make kernel v4.11.0 as the minimum requirement in our documentation?

Thoughts?

Madhu-1 · 2022-02-09T07:15:32Z

@Madhu-1 @nixpanic

Sorry guys! looks like this went into some spam email box or somehow I ignored on this email thread notification may be based on the title. Anyways I will look into this now.

I see the /sys/bus/rbd/supported_features is part of linux kernel v4.11.0
commit 8767b293a4ab6632f9288f34bcf2ab9ba20dca3a
Author: Ilya Dryomov <[email protected]>
Date:   Thu Mar 2 19:56:57 2017 +0100

    rbd: supported_features bus attribute
    
    ... so that userspace can generate meaningful error messages and spell
    out unsupported features that need to be disabled.
    
    Signed-off-by: Ilya Dryomov <[email protected]>
    Reviewed-by: Sage Weil <[email protected]>
So, What route do we want to take here?

The difficulty part is that what should be the minimum sensible values?
RBD_FEATURE_LAYERING is added around linux v3.6, RBD_FEATURE_STRIPINGV2 is added around v3.9, RBD_FEATURE_EXCLUSIVE_LOCK is added around v4.8, and RBD_FEATURE_DATA_POOL around v4.10

As we already have check for deep-flatten, I think it makes it hard to have a kernel version check as we need to maintain both upstream and downstream kernel versions. is there any alternative for this one?

Shoud we implement some intellegence in Ceph-CSI to check kernel version and decide the min suported feature list?

If the file is missing can we continue with starting the driver? and consider this check only if someone asks for auto fallback?

Or should we just make kernel v4.11.0 as the minimum requirement in our documentation?

This breaks things as we never enforced the kernel version from the cephcsi side. and this was never a blocker?

Thoughts?

pkalever · 2022-02-09T07:34:44Z

As we already have check for deep-flatten, I think it makes it hard to have a kernel version check as we need to maintain both upstream and downstream kernel versions. is there any alternative for this one?

The other alternative I can think of is,

If kernel version is < 4.11.0 then just assume we have RBD_FEATURE_LAYERING only, if that doesn't hurt much.
Have a user facing flag to force bypass this feature checks assuming user checked the rbd features list on his own and is sure about the supportability of various features mentioned in the artifacts.

Shoud we implement some intellegence in Ceph-CSI to check kernel version and decide the min suported feature list?

If the file is missing can we continue with starting the driver? and consider this check only if someone asks for auto fallback?

Need to understand where exactly we are working with the features list in the code. But yes, I think continuing to run the Driver looks good. I may get back on this later.

Or should we just make kernel v4.11.0 as the minimum requirement in our documentation?

This breaks things as we never enforced the kernel version from the cephcsi side. and this was never a blocker?

May we just make it a recommendation rather than a requirement?