Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a configuration parameter to tune tasks dispatcher metrics refresh time #715

Closed
laurentganne opened this issue Mar 17, 2021 · 0 comments · Fixed by #716
Closed

Add a configuration parameter to tune tasks dispatcher metrics refresh time #715

laurentganne opened this issue Mar 17, 2021 · 0 comments · Fixed by #716
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@laurentganne
Copy link
Contributor

On a setup with just one deployment, although there is no workflow activity, consul shows an important disk writes activity of 60 K/s.

The command consul monitor -log-level debug | grep PUT | grep _yorc shows yorc acquires one lock per deployment task every second:

16:03:19 [DEBUG] http: Request PUT /v1/session/create?dc=dc1 (1.34792ms) from=127.0.0.1:38196
16:03:19 [DEBUG] http: Request PUT /v1/kv/_yorc/tasks/5c68f529-5db6-419d-9751-ad434166cd9b/.runningExecutionsLock?acquire=4c28b673-dfc6-6876-244e-9b8c93b19185&dc=dc1&flags=3304740253564472344 (939.545µs) from=127.0.0.1:38196
16:03:19 [DEBUG] http: Request PUT /v1/kv/_yorc/tasks/5c68f529-5db6-419d-9751-ad434166cd9b/.runningExecutionsLock?dc=dc1&flags=3304740253564472344&release=4c28b673-dfc6-6876-244e-9b8c93b19185 (861.453µs) from=127.0.0.1:38196
16:03:19 [DEBUG] http: Request PUT /v1/session/destroy/4c28b673-dfc6-6876-244e-9b8c93b19185?dc=dc1 (1.682062ms) from=127.0.0.1:39140

This is coming from the workflow tasks dispatcher taking a lock for each task to compute the number of waiting executions and emit the corresponding metric, every second.
A configuration parameter should be added to tune this time to refresh this dispatcher metrics, with a default value of 5 minutes, to avoid excessive usage of disk writes.

@laurentganne laurentganne added the enhancement New feature or request label Mar 17, 2021
laurentganne added a commit that referenced this issue Mar 19, 2021
@laurentganne laurentganne self-assigned this Mar 22, 2021
@laurentganne laurentganne added this to the 20.05 milestone Mar 22, 2021
laurentganne added a commit that referenced this issue Mar 22, 2021
…er-metrics

Backport40/feature/gh 715 dispatcher metrics
laurentganne added a commit that referenced this issue Mar 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant