Alert Escalation

Introduction

Alert escalation is an action taken on an alert when it is not acknowledged within a desired time period. The alert escalation policy allows an administrator to configure escalation rules which determine.

1. Alerts that need attention

2. The users who are required to act on alerts

3. Action to be taken

Create Alert Escalation Policy

Define Name and Scope

1. Log into OpsRamp.

2. Click All Clients and select the client.

3. From the options in the drop-down menu, click Setup.

4. From the left pane, click Alert Management>Alert Escalation. Alert Escalation Policies list page appears.

5. Click Add, to add a new escalate alert policy.

6. Provide Name and Description.

7. Select the Enabled state from the drop-down list. Following table describes the different states of alert escalation policy:

   Enabled Mode	Description
   ON	Alert escalation policy is created and escalation is performed
   Observed	An escalated alert in Observed mode is only for information purpose. Observed mode enables you see potential alerts that would be escalated by the policy without creating a real alert escalation policy. In Observed mode, an Observed alert is created for each alert that is escalated. You can view these Observed alerts in the Alerts browser. These Observed alerts are indicated with Observed status. You can only perform Close action on an Observed alert, no other action is allowed. The Close action performed on the Observed alert does NOT affect the actual alert.
   OFF	Alert escalation policy is created, but no escalation is performed.

8. Select the organization whose users will receive the escalations from this policy. Example: If you choose partner organization then only partner users can receive escalation.

9. Click Next: Select Resources.

Select Resources

Select the resources whose alerts will match this policy.

1. The policy allows resource selection from one or multiple clients.

   Note:

   • If you chose to escalate alerts to users belonging to a specific client in the previous step, then you can add resources of the selected client only.

   • You can add up to 100 resources to an escalation policy.

2. Resources can be further filtered by Resource Name, Resource Type, Service Group, Device Group or a Site.

   Note: Upon selection of a parent service group or parent device group, all the resources/services in the respective child groups form part of escalation policy. The alerts generated on the resources/services that match the policy conditions get escalated.

3. Click Next: Define Alert Conditions.

Define Alert Conditions

Filter the type of alerts which occur on the previously selected resources. All alerts on the selected resources will match this policy if NO conditions are defined in this section.

1. Add conditions for an alert based on the alert properties listed.

2. Click Add to add multiple conditions.

3. Choose if Any or All of the defined conditions should be applied to filter the alerts.

   • To use regular expressions to filter alerts, see Appendix for examples.

4. Click Next: Define Escalation Rules.

Escalation rules are actions to be taken when an alert is not acknowledged within a desired period. OpsRamp supports Manual and Automatic Escalations.

Define Escalation Rules – Escalate Directly As Needed

Using this option you can select the users whom you want to directly contact on-demand basis. The escalation process is manual and no automatic actions are taken in this scenario.

1. Select the option Escalate directly as needed.

2. To select the users, click Select Users. Select Users\User Groups\Roster\User Group (Distribution List) as the escalation contact.

Define Escalation Rules – Escalate Automatically As Follows

1. Escalate alert policy supports automatic alert actions in the following ways:

   • Send an escalation notification

   • Create an incident from alert

2. Alert Elapsed Timeline: Allows you to configure the time interval to escalate an alert after it is generated.

   • Select Immediately on the timeline to escalate an alert as soon as the alert is generated. Escalate Immediately means to escalate an alert immediately after finishing Alert Correlation and Alert First Response if the given alert has corresponding correlation and first response policies. Alert Correlation takes time to form the cluster. Therefore, escalation algorithm waits for 5 minutes for the cluster to establish before moving the alert to the next phase. If the alert does not qualify for any correlation or first response policies, the algorithm immediately moves the alert to Alert Escalation.

   • Select Wait Minutes/Hours on the timeline to escalate after the alert has elapsed the set time and continues to match the policy conditions.

Escalate as Notification

1. Select Escalate As Notifications to send periodic notifications to users to make sure that an alert is acknowledged.

2. Send Alert Notifications to – Configure the users to whom the notifications to be sent.

   • To select the users, click Select Users. Select Users\User Groups\Roster\User Group (Distribution List) as the recipients.

3. Notification Priority – An escalation notification carries a priority which is used to determine the channel (Email\SMS\Voice) on which notification is delivered.

   • Example: Policy P1 is configured to send Normal priority notifications for all the matching alerts to user A. Since the priority of the notification is Normal, User A prefers to receive all Normal notifications only via email and not via SMS and Voice.

   • See here for configuring the notification preferences.

4. Send Notifications:

   • Repeat Notification Frequency: Configure repeat notification frequency for selected users. The repeat notification is sent to the user as per the Set Repeat Frequency even if an alert is escalated to a higher level, and the user stops receiving notifications after a certain number of repeated notifications.

     Note:

     • Default repeat frequency is 15 minutes and the number of notifications is 2.

     • Minimum number of notifications is 2 and maximum is 10.

   • Alert State Transition: User can escalate alerts based on alert state transition which allow to notify on selected alert state change. For example, user can escalate alerts as notification when alert state changes from Warning to Critical.

     Note: The notification is a one-time notification sent at the time of the state transition.

5. Add Escalation – Adding escalation levels will help you to escalate the alert to next level if it is not acknowledged at an earlier level.

   • A policy can have multiple numbers of escalation notifications.

   • When a level 2 escalation is added to notify users, users in both level 1 and level 2 receive repeated notification according to their repeat notification frequency respectively.

Escalate as Incident

1. Select Escalate As Incident to automatically create an incident from an alert and assign it to the desired user.

   • No further level of escalation can be added after this as the escalation of alert will end as soon as the incident is created.

   • See Learning-Based Alert Escalation to automatically escalate incidents to appropriate groups, priority or category using machine-learning.

2. For New Incidents – Configure the properties of the incident that will be created when an alert’s condition matches this policy.

   • A new incident is created for an alert if there is no open incident existing for the alert.

   • The incident property tokens available in the auto incident form can be used to customize the subject and description of the incident.

3. For Created Incidents – If an open incident (an incident in any state other than Closed) is available for the alert, the incident gets updated instead of creating a new incident.

   • Update Incident – Configure how to update existing incident of an escalated alert. The incident gets updated every time an escalated alert repeats with a state change.

     • Update incident with latest alert description when alert state changes – The latest alert description is appended on the conversation of the incident. No change to the status of the incident is made based on escalated alert’s state.

     • Resolve incident when an alert is healed – The latest alert description is appended on the conversation of the incident. The incident status is changed to resolved when escalated alert’s state changes to OK.

     • Update incident priority based on these rules – The priority of incident is updated as per rule configured on the alert severity change.

   • Notify – Configure which updates to an incident should be notified to users.

     • For every alert update – Send incident notification when any update of escalated alert is appended to the incident.

     • Only when the alert state got changed – Send incident notification only when escalated alert heals and the update is appended to the incident.

   • To customize the incident notifications,

     • Click Setup, on the left-hand side panel, click Service Desk and then click Notifications.

Review

A summary of all sections of the escalate alert policy is available for review. Edit any section as needed. Enable – The policy is enabled once it is saved. The policy can be saved in disabled mode if needed. No escalations will be applied on alerts until the policy is enabled.

Escalated Alerts

The alert details page displays the below information for an escalated alert.

• Matched Escalate Alert Policies: Click Escalate button on the right-hand side panel of alert details page to see the policies that match the alert.

• Upcoming Escalate Alert Action: Displays three upcoming actions of the matching escalate alerts policy.

• Comments: Displays the escalation activities like escalation notification and auto-incident actions per matched policy.

  Note: If an alert escalation policy is in Observed mode, then the Observed alerts show OpsQ recommendation on incident routing, prioritization, categorization etc.

• See Example of an escalation notification.

  • Escalation – Name of the escalate alert policy.

  • Other users notified – List of other users in the other levels of escalation who received the escalation notification.

• The incident details page displays the below information for an auto-incident.

  • Matched Escalate Alert Policies: Click Escalate Alerts on the top header of incident details page to see the policies that match the alert and policy that created the incident automatically.

  • Alerts: Displays the alert that is attached to the auto-incident.

Scenarios

Escalate alerts manually to infrastructure management at the customer site

An organization has outsourced its IT operations management to a service provider. The service provider has an NOC team (Network Operations Center) to address alerts on the organization’s infrastructure. The service provider wants to create contact of escalations from the customer site for the NOC team to escalate alerts on a need basis.

Solution

1. Create users or rosters or user groups to represent the customer team.

2. Configure escalate alerts policy to escalate alerts as needed and assign to the customer team.

Escalate alerts automatically to shift users

A customer has a team of users working in shifts to address alerts on their IT infrastructure. The customer wants to escalate alerts to users available in the active shift at any given time.

Solution

1. Create a roster and add shift details like users, shift time.

2. Configure escalate alerts policy to use rosters to escalate alerts automatically as notifications. See here the instructions for configuring escalate alerts as Notifications.

Escalate alerts from specific devices to specific users

A customer wants to immediately escalate critical alerts on a group of database servers to the database admin team.

Solution

1. Create a roster or user group to represent the database admin team.

2. Create a device group under on the Infrastructure tab, add the member to the database servers.

3. Add device group as a resource from the escalate alerts policy.

4. Configure an alert condition on alert state attribute to filter only critical alerts.

5. Configure an immediate escalation to send a notification as soon as alerts are generated. See here the instructions for configuring immediate escalation.

Escalate alerts from third-party monitoring applications

A customer has integrated third-party alert monitoring application with OpsRamp and wants to escalate all the critical alerts generated from integrated application to their IT administration team.

Solution

1. Create an escalate alerts policy and select All Resources of Partner/Client.

2. Configure an alert condition on Alert Source and Alert State attributes to filter only critical alerts.

   Note: Ensure the match type for alert conditions is All.

3. Configure an immediate escalation to send a notification as soon as the alert is generated. See here the instructions for configuring immediate escalation.

Escalate alerts to multiple users if not acknowledged in a specified time period.

A service provider wants escalation notifications on firm critical applications to be sent to the monitoring team, as soon as the alerts are generated. The service provider has an SLA of 20 minutes to acknowledge these alerts and hence wants the monitoring team to get a reminder every 5 minutes until the alert is acknowledged. Additionally, service provider wants to escalate the alerts to the team manager if the alerts are not acknowledged by the team within SLA.

Solution

1. Add an immediate automatic escalation and configure repeat notification frequency from the escalate alerts policy.

2. Add another escalation with a required time interval and configure escalations.

Receive alert escalation notification via SMS and Voice.

A customer has configured automatic escalations on critical exchange server alerts to a group of administrators. The administrators want to receive the escalations on their mobile devices in addition to an email to be able to follow issues on the exchange server when emails are not accessible.

Solution

1. Contact your partner administrator or OpsRamp support to subscribe and enable SMS\Voice notification service for your organization.

2. Configure notification preferences for SMS and voice based on escalation notification priority under My Profile. See here the instructions for configuring Notification Preferences.

Escalate alerts on specific devices as an incident to a service team.

A customer has an IT application team to address issues on hosted applications. Customer wants all alerts on the business-critical applications to be immediately escalated as incident so that they can be quickly routed to the right assignee in the IT application team.

Solution

1. Create an escalate alerts policy and select the resources on which the applications are hosted.

2. Filter application level alerts by adding alert conditions based on alert metric or alert subject.

3. Configure an immediate escalation as incident and assign the desired incident groups. See here the instructions for configuring escalate alerts as Incident.

Escalate alerts based on alert state change

User has defined a job to install patches on Windows servers. During the installation, alerts in Warning state are triggered. User wants to receive notifications when alert state changes from Warning to Critical.

Solution:

1. Create an escalate alerts policy and select the resources.

2. Filter resource level alerts by adding alert conditions based on Alert State. For example, if you want to receive notification of Critical alerts, select Alert State Is Critical.

3. Configure escalation rules to escalate as Notification, and select the User.

4. In Send Notifications section, select on alert state transition from and then select the alert state.

Escalate alerts which trigger repeatedly

Scenario: A user is monitoring alerts on various SNMP resources which are of high priority. The user wants to escalate an alert to the network administrator if the same alerts occur twice within two days despite Acknowledge/Suppress actions.

Note:

• From the above scenario, the term Same alert indicates alert with the same metric type, resource ID and alert status Critical or Warning.

• Below are two conditions that define an alert occurrence:

  • Change in an alert state from Critical or Warning to OK state.

  • Same alert being triggered repeatedly despite the Suppress action.

Solution:

1. Create an escalate alerts policy by selecting the SNMP resources.

2. In Define Alert Conditions section, select property Alert: Occurrence Frequency, and select values for time range. For the above scenario, select value as 2 Occurrences within 2 Weeks.

3. Configure an immediate escalation as incident and assign the incident to the desired user/user group. See here for instructions to escalate alerts as an incident.

Appendix

Rosters

Roster is a list of users with their schedule of duties within an organization.

1. Login to OpsRamp.

2. Click All Clients and then select the client.

3. On the drop-down menu, click Setup.

4. On the left-hand side panel, click Rosters.

5. Click Add to add a new roster.

6. Provide Name and Description.

7. Shifts Include Users From – organization which the users of shift belong to.

   • A roster can have shift users belong to only a partner organization or only a specific client organization.

     • Provide shift name and schedule details.

     • Select Users – You can select the users from User\User groups.

     • Once users are selected click Select

     • Add a required number of shifts to the roster and click Create.

Notification Preferences

1. OpsRamp supports notification via Email, SMS and Voice channels.

   • See here how to enable SMS and Voice feature for your organization.

   • Currently, the preferences are applicable only for escalation notifications.

2. Users can configure these preferences under their user profile based on notification priority.

   1. Example: Administrator has configured escalation notifications to a user. User can choose to receive notifications on different channels (SMS, Email and Voice) based on notification priority.

Configure Notification Preferences under User Profile

1. Click My Profile.

2. On the left-hand side panel click Preferences and then click Notification Preferences. Notification preference section appears.

3. Click Edit. Configure how the alert notifications should be received and then click Save.

4. Primary Email is the default email used for sending alert notifications. However, you can choose to use an alternative Email.

5. Click Change and then select Alternate Email.

Note: The SMS and Voice notification

1. The SMS and Voice Notification is sent to the Mobile Number available in the user profile. Mobile number format should be prefixed with country code. Example: The mobile number format for united states is [phone].

2. To define mobile number, primary and alternate emails, Click My Profile at the username, and click Edit and then define the mobile number, primary and alternate emails.

Enable SMS and Voice Notification for an organization

To enable SMS and Voice Notification for a partner organization, please contact OpsRamp Support. After the feature is enabled for partner organization, Partner administrator can enable SMS and Voice notification for client organization.

1. Click Setup tab, on the left-hand side panel click Client and then click Edit.

2. Select the option Yes against SMS and Voice notifications and click Finish.

Notification Templates

This template is used to configure the content of Email, SMS, and Voice notification for alerts. The template supports custom tokens to add properties of alert and escalate alert policy in the notification.

1. Click Setup.

2. On the left-hand side panel, click Notifications and then click Alert. Alert templates section appears.

3. Select the notification channel to customize the template.

4. Edit the template as needed. Click on the alert tokens on the left-side panel to add them to the template.

5. Click Preview and then click Save.

Note:

1. To restore the default notification template, click Restore Default and then click Save.

2. SMS text is limited to 1600 characters and messages over 160 characters will be sent as multiple messages.

Change the Mode of Alert Escalation Policy

You can change the mode of alert escalation policy to ON or Observed or OFF modes. See here for more information on these nodes.

Perform these steps:

1. On the Alert Escalation Policies page, select the required policy.

2. In Enabled drop-down, select the required mode. The enabled mode is changed.