Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: missing results in ListObjects/CommonPrefixes #87

Merged
merged 1 commit into from
Jan 11, 2021
Merged

fix: missing results in ListObjects/CommonPrefixes #87

merged 1 commit into from
Jan 11, 2021

Conversation

justenwalker
Copy link
Contributor

@justenwalker justenwalker commented Jan 9, 2021

When listing blobs, each page may have up to the max results (default 5000). However: you may still get less than that amount. To get all results, azure requires us to paginate through them using NextMarker/Marker until NextMarker is unset in the response. Currently only the first page is ever considered, since this is called only once.

This fix affects the Backup Sync Controller which uses these to list the available backups and delete orphaned backups if they are not found in storage. When providing an incomplete list of results, the controller erroneously removes backup resources from the cluster when they still exist in storage.

Signed-off-by: Justen Walker [email protected]

Paginate the results using NextMarker/Marker so that all results are
returned instead of potentially only a subset of objects.

When listing blobs, each page may have up to the max results (default 5000).
However: you may still get less than that amount. To get all results, azure
requires us to paginate through them using NextMarker/Marker until NextMarker
is unset in the response. Currently only the first page is ever considered,
since this is called only once.

This fix affects the Backup Sync Controller which uses these to list the
available backups and delete orphaned backups if they are not found in storage.
When providing an incomplete list of results, the controller erroneously
removes backup resources from the cluster when they still exist in storage.

Signed-off-by: Justen Walker <[email protected]>
Copy link
Contributor

@zubron zubron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for investigating this and providing a fix! 👍

Copy link
Contributor

@carlisia carlisia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 thanks for this fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants