Brief blips are a fact of life when it comes to monitoring, and you don't want to be woken up in the middle of the night by them just to find that nothing of importance has happened. Time series monitoring also comes with many minor race conditions that can cause a alert to fire spuriously for a short time period.

The for clause of an alert specifies how long a time series must appear in an alert expression before the alert is considered firing and sent on to the Alertmanager. While waiting for the for clause, the alert will be in the pending state in Prometheus.

It is recommended to always have a for clause of at least 5 minutes, as this is a good balance between avoiding noisy paging and not seeing problems fast enough. Keep in mind that it usually takes around 5 minutes for an oncall engineer to get to their laptop and start debugging.

The for state currently only lives in memory of the Prometheus, so restarts of the Prometheus will reset alerts. It is recommended to avoid values of over an hour.