Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v10.4.x] Alerting: Make context deadline on AlertNG service startup configurable #96058

Merged
merged 5 commits into from
Nov 7, 2024

Conversation

fayzal-g
Copy link
Contributor

@fayzal-g fayzal-g commented Nov 7, 2024

Backport 1fdc48f from #96053


What is this feature?
Adds a new field unified_alerting.initialization_timeout that allows the context deadline for the AlertNG service to be configurable. The default timeout is 30s which is the current hardcoded value, but can be extended in cases where users are hitting this timeout, which can be seen in cases where there are a high number of orgs.

Who is this feature for?
Users that are hitting errors such as:

Unable to create Alertmanager for org    logger=ngalert.multiorg.alertmanager org=1 error="error reading file 'notifications' from database: context deadline exceeded"

Which issue(s) does this PR fix?:
Fixes #69080

Special notes for your reviewer:

Please check that:

  • It works as expected from a user's perspective.
  • If this is a pre-GA feature, it is behind a feature toggle.
  • The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

fayzal-g and others added 2 commits November 7, 2024 18:46
…le (#96053)

* Make alerting context deadline configurable

* Remove debug logs

* Change default timeout

* Update tests

(cherry picked from commit 1fdc48f)
@fayzal-g fayzal-g added area/backend add to changelog backport A backport PR product-approved Pull requests that are approved by product/managers and are allowed to be backported labels Nov 7, 2024
@fayzal-g fayzal-g added this to the 10.4.x milestone Nov 7, 2024
@fayzal-g fayzal-g requested review from torkelo and a team as code owners November 7, 2024 18:50
@fayzal-g fayzal-g requested review from papagian, zserge and undef1nd and removed request for a team November 7, 2024 18:50
@fayzal-g fayzal-g merged commit 880e332 into v10.4.x Nov 7, 2024
9 checks passed
@fayzal-g fayzal-g deleted the backport-96053-to-v10.4.x branch November 7, 2024 19:25
grafana-delivery-bot bot pushed a commit that referenced this pull request Nov 8, 2024
…configurable (#96058)

* Alerting: Make context deadline on AlertNG service startup configurable (#96053)

* Make alerting context deadline configurable

* Remove debug logs

* Change default timeout

* Update tests

(cherry picked from commit 1fdc48f)

* remove file

* Remove go.work.sum change

* Fix backport

* No cyclo flag

(cherry picked from commit 880e332)
@aangelisc aangelisc modified the milestones: 10.4.x, 10.4.12 Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
add to changelog area/backend backport A backport PR product-approved Pull requests that are approved by product/managers and are allowed to be backported
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants