Velero in Azure - Is it possible to store the backups with redundancy between two different AZs?

[BCMBCM]

I am aware Velero is not able to backup Azure ZRS managed disks, but I would like to know if it is possible to store the pvc backup snapshots in ZRS target locations and how to do configure it (using velero plugin for Microsoft Azure)

Regarding the backups stored in the Blob STA, I understand it would just be a matter of deploying the Blob STA as ZRS. Is this right?

PS: Please can you label this issue as a question?

Thank you

[blackpiglet]

You can change the StorageClass used in restore process. This is the document link: https://velero.io/docs/v1.8/restore-reference/#changing-pvpvc-storage-classes

[BCMBCM]

@blackpiglet the question is not related to the restore process, it is about having more resilience of the backups stored in Azure
@ywk253100 Do you know if it is possible to configure velero to store the snapshots with Zone Redundant Storage?

Thank you

[ywk253100]

@BCMBCM
Backing up ZRS managed disks should already be fixed by vmware-tanzu/velero-plugin-for-microsoft-azure#131

Azure Velero plugin should support storing the backup in ZRS location, but we didn't test on it. It will be great if you have the chance to try that and see if it works as expected

[BCMBCM]

@ywk253100 Sorry If i haven´t explained it well. I am not trying to backup ZRS disks. What I would like to do is to be able to store my backup snapshots in a ZRS storage , to provide more resilence to those images.
Is it possible or are plans to add it to velero?

Thank you

[chreichert]

@ywk253100 Is there anything happening here? I would even declare this as feature request.
For redundancy reasons it would be very much appreciated, when full disk snapshots would be stored in ZRS mode instead of LRS as it is current behaviour.
Thanks.

[reasonerjt]

@anshulahuja98 Could you answer this question?

[anshulahuja98]

All managed disk snapshots in zonal regions are created as ZRS by default.

https://learn.microsoft.com/en-us/azure/virtual-machines/disks-incremental-snapshots?tabs=azure-cli
Additionally, incremental snapshots potentially offer better reliability with [zone-redundant storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-redundancy) (ZRS). If ZRS is available in the selected region, an incremental snapshot will use ZRS automatically. If ZRS isn't available in the region, then the snapshot will default to [locally-redundant storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-redundancy) (LRS). You can override this behavior and select one manually but, we don't recommend that.

I am not fully sure if Azure Plugin might be specifically requesting for LRS snapshots, my hunch will be that snapshots in zonal regions will be ZRS by default.

We have validated this in CSI Plugin based snapshotting. But for Azure Plugin based snapshotting you can perhaps give inputs on what default behaviour you are seeing.

[anshulahuja98]

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/85299d0088599b1b134d46776fe8faa2d50ae470/velero-plugin-for-microsoft-azure/volume_snapshotter.go#L251C1-L263C1

I also checked the code, doesn't seem like anything explicit is being set and hence snapshots must be ZRS by default, let us know if you see behaviour otherwise.

[anshulahuja98]

Would request @chreichert to share some actual snapshots they have seen as part of backup in zonal regions as LRS as datapoint for investigation.
I would pick up POC post that.

[chreichert]

@anshulahuja98 We recently switched to incremental backups after reading the docs again and understanding, that incremental backups always will be ZRS. So unfortunately I am not able to share snapshots to verify, that they are LRS even in a ZRS region.
But our observed behaviour before was, that snapshots will be written to LRS storage when using full-backup mode for the snapshots, which is the default setting if not specified otherwise. After explicitly setting incremental backup mode for the snapshots (configuration.volumeSnapshotLocation[0].config.incremental=true) snapshots were written to ZRS storage instead of LRS.
We also found somewhere in the docs, that Azure is automatically taking care of the snapshots during backup-expiry, so that data is recoverable at any time, even if older backups/snapshots have been deleted after expiry time.
For us the issue is solved, using incremental backup mode for the snapshots.

[anshulahuja98]

Yes perfect. Incremental is the recommendation.
Will resolve the issue for now.

[chreichert]

@anshulahuja98 Maybe some addition in the Azure Plugin docs would be helpful, that one should use incremental mode if ZRS snapshots are desired, as default behaviour is to use full backup mode.

[anshulahuja98]

Valid suggestion. I'll do that.