Crisis

Since someone asked in the nagios users mailing list for an help to understand how notification escalations work in Nagios, I wrote an easy and short workflow to explain the logic underneath.

Notification escalations are a means to ensure that someone will ne notified, eventually. If a contact doesn’t acknowledge a problem, we can escalate the nofications to someone else, changing the way we deliver the messages if we think it’s a good strategy.

In this post we won’t discuss of the syntax or the options regarding the escalations, we will have a look at the logic of the escalations, at how they take place and how they work.

Here the workflow for check/notification/escalation is the following:

1. The service/host is checked in OK state with the check_interval timing;
2. As the service/host goes into a NON OK state, but didn’t reach the max_check_attempts, the service enters a SOFT NON OK state and the next check is scheduled with the retry_interval timing;
3. As the service/host in NON OK state reaches the max_check_attempts value, the service enters an HARD NON OK STATE and the next service/host check is scheduled with the check_interval timing;
4. Now, if you set first_notification_delay, this can alter the timing for the first notification to be sent (0 means notifications to be sent immediately);
5. If you didn’t set the first_notification_delay, the first notification is immediately sent and the following will be scheduled with the notification_interval timing (0 means only the first notification will be sent, the other will not be sent);

Now, let’s make a practical example:

define serviceescalation{

host_name               webserver

service_description     HTTP

first_notification      3

last_notification       5

notification_interval   45

contact_groups          ITOps_Oncall,managers

}
define serviceescalation{

host_name               webserver

service_description     HTTP

first_notification      6

last_notification       0

notification_interval   60

contact_groups	  ITOps_Oncall,managers,everyone

}

What happens? Here we go:

1. In this escalation, at the third notification (with the notification_interval taking place), the notification interval changes to 45 minutes, so the first notification will be sent after the max_check_attempts value will be reached (assuming you didn’t put any delay), then the second after 10 time units, usually 10 minutes, the third 10 mins after the second, the fourth 45 mins after the third, the fifth 45 mins after the fourth, the sixth 45 mins after the fifth.
2. From the sixth notification, the new escalation comes into play. The seventh notification will be sent after 60 minutes, and all the other notifications will be sent 60 mins after each other. Keep in mind that having used 0 as the last_notification value, you escalation will never end till your check will return an OK status.