Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing to mount filesystem on rbd device #3913

Closed
Rakshith-R opened this issue Jun 19, 2023 · 3 comments
Closed

Failing to mount filesystem on rbd device #3913

Rakshith-R opened this issue Jun 19, 2023 · 3 comments
Labels
bug Something isn't working component/rbd Issues related to RBD wontfix This will not be worked on

Comments

@Rakshith-R
Copy link
Contributor

This seems to be intermittent.

Failed e2e https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/organizations/jenkins/mini-e2e-helm_k8s-1.26/detail/mini-e2e-helm_k8s-1.26/381/pipeline/109

NodeStage call: https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e-helm_k8s-1.26/runs/381/nodes/100/steps/103/log/?start=0

 I0616 12:48:57.100748   54200 utils.go:195] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 GRPC call: /csi.v1.Node/NodeStageVolume
  I0616 12:48:57.100861   54200 utils.go:206] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["discard"]}},"access_mode":{"mode":7}},"volume_context":{"clusterID":"f56081c2-a7cf-4225-8774-8e02c2ff0c44","encrypted":"true","encryptionKMSID":"vault-test","encryptionType":"block","imageFeatures":"layering","imageName":"csi-vol-3783587d-7209-4bba-984c-46cbe402dbb9","journalPool":"replicapool","pool":"replicapool","storage.kubernetes.io/csiProvisionerIdentity":"1686919157186-759-rbd.csi.ceph.com"},"volume_id":"0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9"}
  I0616 12:48:57.102103   54200 omap.go:88] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 got omap values: (pool="replicapool", namespace="", name="csi.volume.3783587d-7209-4bba-984c-46cbe402dbb9"): map[csi.imageid:12c8f92f085d csi.imagename:csi-vol-3783587d-7209-4bba-984c-46cbe402dbb9 csi.volname:pvc-7dab02e8-f225-478a-9a61-331502529a57 csi.volume.encryptKMS:vault-test csi.volume.encryptionType:block csi.volume.owner:rbd-65]
  I0616 12:48:57.148262   54200 rbd_util.go:352] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 checking for ImageFeatures: [layering]
  I0616 12:48:57.178248   54200 cephcmds.go:105] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 command succeeded: rbd [device list --format=json --device-type krbd]
  I0616 12:48:57.191442   54200 rbd_attach.go:419] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 rbd: map mon rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789
  I0616 12:48:57.257866   54200 cephcmds.go:105] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 command succeeded: rbd [--id cephcsi-rbd-node -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 --keyfile=***stripped*** map replicapool/csi-vol-3783587d-7209-4bba-984c-46cbe402dbb9 --device-type krbd --options noudev]
  I0616 12:48:57.257930   54200 nodeserver.go:425] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 rbd image: replicapool/csi-vol-3783587d-7209-4bba-984c-46cbe402dbb9 was successfully mapped at /dev/rbd0
  I0616 12:48:57.271590   54200 encryption.go:88] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 image replicapool/csi-vol-3783587d-7209-4bba-984c-46cbe402dbb9 encrypted state metadata reports "encrypted"
  I0616 12:48:57.277153   54200 crypto.go:320] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 "/dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9" is not an active LUKS device (an error (exit status 4) occurred while running cryptsetup args: [status luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9]): 
  I0616 12:48:57.277186   54200 crypto.go:272] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 Opening device "/dev/rbd0" with LUKS on "luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9"
  E0616 12:48:59.315919   54200 crypto.go:275] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 failed to open device "/dev/rbd0" (<nil>): DM-UUID for device luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 was truncated.
  I0616 12:48:59.316044   54200 mount_linux.go:579] Attempting to determine if disk "/dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9])
  I0616 12:48:59.321990   54200 mount_linux.go:582] Output: "DEVNAME=/dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9\nTYPE=jmicron_raid_member\n"
  I0616 12:48:59.322048   54200 mount_linux.go:579] Attempting to determine if disk "/dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9])
  I0616 12:48:59.328100   54200 mount_linux.go:582] Output: "DEVNAME=/dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9\nTYPE=jmicron_raid_member\n"
  W0616 12:48:59.328137   54200 mount_linux.go:533] Configured to mount disk /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 as jmicron_raid_member but current format is ext4, things might break
  I0616 12:48:59.328160   54200 mount_linux.go:441] Checking for issues with fsck on disk: /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9
  I0616 12:48:59.335271   54200 mount_linux.go:546] Attempting to mount disk /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9
  I0616 12:48:59.335296   54200 mount_linux.go:220] Mounting cmd (mount) with arguments (-t ext4 -o _netdev,discard,defaults /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9)
  E0616 12:48:59.342669   54200 mount_linux.go:232] Mount failed: exit status 32
  Mounting command: mount
  Mounting arguments: -t ext4 -o _netdev,discard,defaults /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9
  Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9: wrong fs type, bad option, bad superblock on /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9, missing codepage or helper program, or other error.

  E0616 12:48:59.342712   54200 nodeserver.go:832] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 failed to mount device path (/dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9) to staging path (/var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9) for volume (0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9) error: mount failed: exit status 32
  Mounting command: mount
  Mounting arguments: -t ext4 -o _netdev,discard,defaults /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9
  Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9: wrong fs type, bad option, bad superblock on /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9, missing codepage or helper program, or other error.
   Check dmesg logs if required.
  I0616 12:48:59.349974   54200 crypto.go:294] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 Closing LUKS device "luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9"
  I0616 12:48:59.407299   54200 cephcmds.go:105] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 command succeeded: rbd [unmap /dev/rbd0 --device-type krbd --options noudev]
  E0616 12:48:59.407420   54200 utils.go:210] ID: 244 Req-ID: 0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 GRPC error: rpc error: code = Internal desc = mount failed: exit status 32
  Mounting command: mount
  Mounting arguments: -t ext4 -o _netdev,discard,defaults /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9 /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9
  Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/rbd.csi.ceph.com/10620984cfc177b4ba16a0470c428a6fde19de97eeb6c54a19dc58aad25e8f12/globalmount/0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9: wrong fs type, bad option, bad superblock on /dev/mapper/luks-rbd-0001-0024-f56081c2-a7cf-4225-8774-8e02c2ff0c44-0000000000000002-3783587d-7209-4bba-984c-46cbe402dbb9, missing codepage or helper program, or other error.

dmesg log: https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e-helm_k8s-1.26/runs/381/nodes/109/log/?start=0

* ==> dmesg <==
* [  +0.007023]   #5
[  +0.000966] cpu 5 spinlock event irq 83
[  +0.006031]   #6
[  +0.000968] cpu 6 spinlock event irq 89
[  +0.006039]   #7
[  +0.000976] cpu 7 spinlock event irq 95
[  +0.230960] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[  +0.216389] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
              * this clock source is slow. Consider trying other clock sources
[Jun16 10:17] Grant table initialized
[  +0.004336] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[  +3.002119] printk: systemd: 17 output lines suppressed due to ratelimiting
[  +3.648388] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[  +0.360430] cirrus 0000:00:02.0: [drm] drm_plane_enable_fb_damage_clips() not called
[Jun16 12:47] EXT4-fs (dm-0): mounting with "discard" option, but the device does not support discard
[ +20.362384] EXT4-fs (dm-0): mounting with "discard" option, but the device does not support discard
[Jun16 12:48] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[  +2.864178] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[Jun16 12:49] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[  +4.459215] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[  +6.394195] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[ +10.381713] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[ +18.426151] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[Jun16 12:50] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[Jun16 12:51] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[Jun16 12:53] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[Jun16 12:55] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[Jun16 12:57] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[Jun16 12:59] ceph: mds0 session blocklisted
[ +10.742515] ceph: mds0 session blocklisted
[ +10.168918] ceph: mds0 session blocklisted
[  +8.881455] ceph: mds0 session blocklisted
[Jun16 13:00] ceph: mds0 session blocklisted
[  +9.283747] ceph: mds0 session blocklisted
[Jun16 13:02] ceph: mds0 session blocklisted
[ +10.076713] ceph: mds0 session blocklisted
[ +42.753927] ceph: mds0 session blocklisted
[Jun16 13:03] ceph: mds0 session blocklisted
[ +14.305634] ceph: mds0 session blocklisted
[  +6.060077] ceph: mds0 session blocklisted
[ +10.079681] ceph: mds0 session blocklisted
[Jun16 13:04] ceph: mds0 session blocklisted
[ +10.691762] ceph: mds0 session blocklisted
[Jun16 13:05] ceph: mds0 session blocklisted
[ +10.288410] ceph: mds0 session blocklisted
[Jun16 13:06] ceph: mds0 session blocklisted
[Jun16 13:07] ceph: mds0 session blocklisted
[ +21.751703] ceph: mds0 session blocklisted
[Jun16 13:08] ceph: mds0 session blocklisted
[ +24.104066] ceph: mds0 session blocklisted
[  +6.971149] ceph: mds0 session blocklisted
[ +16.012434] ceph: mds0 session blocklisted
[Jun16 13:09] ceph: mds0 session blocklisted
[ +15.941840] ceph: mds0 session blocklisted
[ +24.508792] ceph: mds0 session blocklisted
[Jun16 13:10] ceph: mds0 session blocklisted
[ +12.088382] ceph: mds0 session blocklisted
[ +16.242294] ceph: mds0 session blocklisted
[Jun16 13:11] ceph: mds0 session blocklisted
[ +16.039934] ceph: mds0 session blocklisted
@Rakshith-R Rakshith-R added the component/rbd Issues related to RBD label Jun 19, 2023
@Rakshith-R Rakshith-R mentioned this issue Jun 19, 2023
Merged
@Rakshith-R Rakshith-R added the bug Something isn't working label Jun 19, 2023
@nixpanic
Copy link
Member

The logs are gone... Note that the mds0 messages in the logs are CephFS related, and not RBD.

It seems that the problem happened with an encrypted rbd-image, which was mapped and decrypted successfully. While mounting, there is no ext4 filesystem detected, so potentially the formatting of the device went wrong?

Maybe mounting with discard caused a failure, as that option is not supported by the underlying storage? However, I expect that is only a warning and not a fatal error.

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the wontfix This will not be worked on label Aug 16, 2023
@github-actions
Copy link

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working component/rbd Issues related to RBD wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nixpanic @Rakshith-R