Adding an alert mechanism into lvm-operator

[aruniiird]

Adding an alert mixins framework into LVM Operator.
Makefile is modified to generate the prometheus alert yaml file.

Added 'VolumeGroupUsageAtThreshold' alert which will be triggered if
volumegroup usage goes beyond the threshold percentage (default set to 75%)

Signed-off-by: Arun Kumar Mohan [email protected]

[nbalacha]

Is it possible to just create a static YAML file for this?

Hit approve by mistake.

[nbalacha]

Please update the commit msg to match the commitlint requirements

[iamniting]

After discussing This with @aruniiird I understand that we can create/update the YAML files using the jsonnet tool via just changing the values in the monitoring/config.libsonnet file instead of changing the YAML file directly if we require any change in the future. Also, we can create different YAML files too using this tool for different use cases eg. Managed Services, snow, etc.

So the question to @nbalacha is do we really want to have that flexibility? if not we can create a static file too as you suggested.

If we are gonna stick to the same format we need to make sure that

• Make target also download jsonnet tool if not already present on the machine.
• We should use kustomize to use metadata from a different file and spec from a different file as this tool just create only spec and not metadata at all.
• We should also upload the generated YAML file.

[umangachapagain]

Suggested change

	message: 'VolumeGroup {{ $labels.device_class }} utilization has crossed 75% percent on node {{ $labels.node }}. Free up some space or expand the VolumeGroup.',
	message: 'VolumeGroup {{ $labels.device_class }} utilization has crossed %(vgUsageThresholdNearFull)0.2f percent on node {{ $labels.node }}. Free up some space or expand the VolumeGroup.',

[aruniiird]

Done

[umangachapagain]

Suggested change

	message: 'VolumeGroup {{ $labels.device_class }} utilization has crossed 85% percent on node {{ $labels.node }}. Free up some space or expand the VolumeGroup immediately.',
	message: 'VolumeGroup {{ $labels.device_class }} utilization has crossed %(vgUsageThresholdCritical)0.2f percent on node {{ $labels.node }}. Free up some space or expand the VolumeGroup immediately.',

[aruniiird]

Done

[umangachapagain]

Suggested change

severity_level: 'error',

[aruniiird]

Done

[umangachapagain]

Suggested change

severity_level: 'warning',

[aruniiird]

Done

[umangachapagain]

Complexity of generating alerts using mixin seems to be more than the complexity of maintaining a single YAML file. So, for now I'd suggest using just a single static YAML file.
Once it gets fairly complex to manage, you can move to mixins or some other solutions.

[agarwal-mudit]

/cherry-pick release-4.10

[openshift-cherrypick-robot]

@agarwal-mudit: once the present PR merges, I will cherry-pick it on top of release-4.10 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nbalacha]

Complexity of generating alerts using mixin seems to be more than the complexity of maintaining a single YAML file. So, for now I'd suggest using just a single static YAML file. Once it gets fairly complex to manage, you can move to mixins or some other solutions.

We did discuss whether we should just use a static YAML and decided to put that effort in now in order to make it easier in the future.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aruniiird, nbalacha, umangachapagain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nbalacha]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-cherrypick-robot]

@agarwal-mudit: new pull request created: #121

In response to this:

/cherry-pick release-4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.