Alerting: Do not store series values from past evaluations in state manager for no reason

[alexweav]

What is this feature?

States in the state manager store the results of several prior evaluations. This includes values, which is the measured dataframe results for that series:

grafana/pkg/services/ngalert/state/manager.go

Lines 361 to 368 in 6e4d35e

	currentState.Results = append(currentState.Results, Evaluation{
	EvaluationTime: result.EvaluatedAt,
	EvaluationState: result.State,
	Values: NewEvaluationValues(result.Values),
	Condition: alertRule.Condition,
	})
	currentState.LastEvaluationString = result.EvaluationString
	currentState.TrimResults(alertRule)

But, the only code that ever reads this field, only looks at the most recent 1 evaluation. Any values in that slice beyond the last one are stored for no reason:

grafana/pkg/services/ngalert/state/state.go

Lines 467 to 471 in 6e4d35e

	if len(a.Results) <= 0 {
	return nil
	}
	lastResult := a.Results[len(a.Results)-1]

The state manager is storing every measured value for every series in every query, including intermediate dataframes. Basically, all queried dataframes for all rules, for multiplpe evaluations in the past, needlessly.

The length of the history is based on For - and can be raised arbitrarily high with a long for. Even for rules with a for of 0, the length is hardcoded to 10 evaluations. For a typical rule with 3 query nodes, this results in 30 frames worth of series kept in memory (spread across states) when we only use 3 of them.

By creating a long For with many dimensions, you can write a rule that eventually consumes arbitrary memory and OOMs grafana provided the process does not restart.

This is a truly incredible amount of data in some cases on instances even with simpler alerting usage it can quickly dominate the entire memory space of unified alerting.

Why do we need this feature?

Likely, most memory consumption of unified alerting across the board is occupied by this throwaway data.

This PR simply stores the most recent single evaluation, rather than a running slice of them. The result is transparent to users, but we stop hanging on to data that we won't use.

Which issue(s) does this PR fix?:
n/a

Special notes for your reviewer:

Please check that:

• It works as expected from a user's perspective.
• If this is a pre-GA feature, it is behind a feature toggle.
• The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

[alexweav]

/deploy-to-hg

[ephemeral-instances-bot]

• Preparing your instance. A comment containing your instance's url will be added to this PR when the instance is ready.
• Your instance will be ready in ~10 minutes. Follow the workflow progress
• Slack channel: #proj-ephemeral-hg-instances
• Building instance with alexweav/do-not-store-old-results oss branch and main enterprise branch. How to choose a branch

[ephemeral-instances-bot]

• Your instance can be accessed at: https://ephemeral1511182187525alexwea.grafana-dev.net
• The instance is not using the CDN assets.
• How to access / How to update instance config / How to build a specific branch

[yuri-tceretian]

Nice improvement. It's long overdue! LGTM

[JacobsonMT]

LGTM 🚀