Alert on Alert Insights Report Data

Alert Insights helps you identify alert noise and operational trends across your environment using your own alert history. In addition to reviewing these trends in the Alert Insights Report, you can create Alerts v2 definitions that notify you when key Alert Insights Report metrics deviate from their normal baseline.

Alert Insights alerts use a rolling baseline to evaluate changes in alert volume, resolution time, alert noise, and open alert backlog.

For information about the underlying report, see Alert Insights Report.
 

Current limitations

• Only Email notification channels are supported.
• Maintenance windows are not supported. Alert Insights alerts continue to evaluate and generate notifications during maintenance windows.
• Alert dismissal behavior depends on how the alert definition is configured:
  • If the alert definition uses No Clear – Clear by Dismissing, dismissing the alert clears it.
  • If the alert definition includes a clear condition, dismissing the alert has no effect. The alert remains active until the configured clear condition is met.
• Alert site selection does not support selecting only the global site or only all multi-sites.

What can I alert on?

You can create Alerts v2 definitions for the following Alert Insights metrics:

Alert Insights metrics are calculated using a rolling 30-day window. Each metric is evaluated against a baseline calculated as the average of the six preceding 30-day periods.

Alert Insights metrics are updated daily.

Sample use cases

The following examples illustrate common ways to use Alert Insights alerts to proactively identify changes in your alerting environment.

Detect an unusual increase in Critical or Emergency alerts

Use an Alert Volume Deviation alert to identify when high severity alert activity is significantly above normal.

For example:

Critical alerts increased 15% over your 30 day rolling baseline (42 vs. normal ~36).

This can help identify an emerging infrastructure issue or an alert definition that has unexpectedly begun generating additional alerts.

Identify when incidents are taking longer to resolve

Use a Resolution Time Degradation alert to detect when median alert resolution times begin increasing.

For example, a notification could indicate:

Median Critical alert resolution time increased 25% (4.2 hours vs. normal 3.4 hours).

This can help identify situations where:

• Incident volume is overwhelming available resources.
• Critical alerts are remaining unresolved longer than expected.
• Operational processes may require investigation.

Detect increasing alert fatigue

Use a Noise Ratio Deviation alert to identify when a growing percentage of alerts are considered candidates for noise reduction.

For example, a notification could indicate that 62% of alerts are candidates for noise reduction.

When you receive this notification, review the Alert Insights Report to identify noisy alert definitions and evaluate whether adding an Alerts v2 delay would reduce unnecessary notifications.

Monitor a growing alert backlog

Use an Open Alert Backlog Deviation alert to identify when unresolved alerts are accumulating faster than normal.

For example:

You have 28 open alerts, 40% above your baseline of 20. Eight are Critical, up from a normal value of three.

A growing backlog can indicate that your team is falling behind on incident response or that an infrastructure issue requires attention.

Create an Alert Insights alert

Create Alert Insights alerts using the standard Alerts v2 workflow.

1. Navigate to Manage Alerts > Alerts.
2. Click Create New Alert.
3. Select an Alert Insights trigger condition.
4. Configure the deviation threshold that triggers the alert.
5. Configure the alert severity and other alert settings.
6. Select an Email notification channel.
7. (Optional) Customize the trigger and clear message using Alert Insights notification variables.
8. Enable the alert.
9. Save the alert.

Current limitations

• Only Email notification channels are supported.
• Maintenance windows are not supported. Alert Insights alerts continue to evaluate and generate notifications during maintenance windows.
• Alert dismissal behavior depends on how the alert definition is configured:
  • If the alert definition uses No Clear – Clear by Dismissing, dismissing the alert clears it.
  • If the alert definition includes a clear condition, dismissing the alert has no effect. The alert remains active until the configured clear condition is met.
• Alert site selection does not support selecting only the global site or only all multi-sites..

Customize email notifications

Alert Insights provides notification variables that let you include current values, baseline values, and deviation percentages directly in email notifications.

Available variables include:

• Current and baseline alert counts by severity
• Alert volume deviation percentage by severity
• Current and baseline median resolution times by severity
• Median resolution time deviation percentage by severity
• Current and baseline noise ratio
• Noise ratio deviation
• Current and baseline open alert count by severity
• Open alert volume deviation percentage by severity

Including these variables provides recipients with immediate context without requiring them to open the Alert Insights Report.

For example:

Critical alerts increased 15% over your 30 day rolling baseline (42 vs. normal ~36).

or

Median Critical alert resolution time increased 25% (4.2 hours vs. normal 3.4 hours).

Best practices

• Configure thresholds that represent meaningful operational changes rather than small fluctuations.
• Consider using lower thresholds for Emergency and Critical alerts than for Warning or Informational alerts.
• Use Alert Insights alerts as an early warning signal, then investigate the underlying cause in the Alert Insights Report.
• If noise ratio alerts increase, review the recommended Alerts v2 delays and the alert definitions contributing the most noise.
• Remember that Alert Insights compares activity against your organization's own baseline, making alerts adaptive to your environment rather than relying on fixed thresholds.