Grafana. Configuring notifications

You can configure notifications to users about the status of virtual machines (VMs) and cluster nodes. Grafana allows you to send notifications by email and/or to various messengers (e.g. Slack or Telegram).

Read more about configuring notifications in Grafana in the official documentation.

Notification channels

Telegram

To configure the notifications channel in Telegram:

1. Enter Alerting → Notification channels.
   
   
2. Press Add channel (New Channel).
3. Specify the notification Name. For example, Telegram notify.
4. In the Type field, select the notification type — Telegram.
5. Enable the option Default (send on all alerts) to use this method for all notifications.

6. Enable the option Include image to add the dashboard image to the notification.

   Note

   We do not recommend using this option, as creating an image places high load on the Grafana docker container.

7. Enable the option Disable Resolve Message so as not to receive a resolve message for an issue regarding which notifications have been sent.

8. Enable the option Send Reminders to receive reminders of the notification created and specify the frequency of sending in the Send reminder every parameter. For example, 15m — for sending the reminder every 15 minutes.
   

   Note

   Reminders cannot be sent more frequently than the notifications themselves.
9. Specify the Telegram settings:
   
   1. BOT API Token — Telegram bot token.
   2. Chat ID — Telegram chat id.
10. Press Send Test to send a test message.
11. Press Save to save the changes.

Example of settings

Slack

To configure the notifications channel in Slack:

1. Enter Alerting → Notification channels.
   
2. Press Add channel (New Channel).
3. Specify the notification Name. For example, Slack notify.
4. In the Type field, select the notification type — Slack.
5. Enable the option Default (send on all alerts) to use this method for all notifications.

6. Enable the option Include image to add the dashboard image to the notification.

   Note

   We do not recommend using this option, as creating an image places high load on the Grafana docker container.

7. Enable the option Disable Resolve Message so as not to receive a resolve message for an issue regarding which notifications have been sent.

8. Enable the option Send Reminders to receive reminders of the notification created and specify the frequency of sending in the Send reminder every parameter. For example, 15m — for sending the reminder every 15 minutes.
   

   Note

   Reminders cannot be sent more frequently than the notifications themselves.

9. Specify the Slack settings:
   

   1. Url — URL address of incoming Slack webhooks. Read more in the official Slack documentation.

   2. Recipient — message recipient. Enter the channel name in the format #channel_name or user name in the format @username.

   3. Username — notification sender user name.
      
   4. Icon emoji — emoji for the notification sender icon. For example, :exclamation:.
      
   5. Icon URL — URL with the notification sender icon image.
      
   6. Mention — Slack users to be mentioned in the message. You can list the names separated by commas. For example, "@peter,@john,@paul".
10. Press Send Test to send a test message.
11. Press Save to save the changes.

Example of settings

Notification parameters

To configure the notification sending rules:

1. Press the graph header → Edit.
   
   
2. Press  → Create alert.
   
3. Specify the notification parameters:
   
   1. Name — discretionary notification name.
   2. Evaluate every — notification interval. For example, 5m for notification every five minutes or 1h for notification every hour.
   3. For — the time that Grafana waits after the threshold value of the parameter has been reached. For example, 15m for 15 minutes waiting time.

   4. Conditions — conditions for notification:

      Condition format

      WHEN function OF query(id, start_time, end_time) condition value

       Comments to the format

      function — function for parameter calculation. Possible values:

      • avg() — average value;
      • min() — minimum value;
      • max() — maximum value;
      • sum() — sum of all values;
      • count() — number of values;
      • last() — last value;
      • median() — median value;
      • diff() — difference of values;
      • percent_diff() — percentage difference of values;
      • count_non_null() — number of values other than null;

      id — queue id. Default value for the first graph — A, for the second — B etc. You can view the id in the   (Queries) section;

      start_time — calculation start time. The start_time parameter is set relative to the current time. For example, 15m — 15 minutes before the current time;

      end_time — calculation end time. The end_time parameter is set relative to the current time in the now-<time> format. For example, now-5m — 5 minutes before the current time;

      condition — condition. Possible values:

      • IS ABOVE — higher;
      • IS BELOW — lower;
      • IS OUTSIDE RANGE — out of range;
      • IS WITHIN RANGE — within range;
      • HAS NO VALUE — no value is available;

      value — value or range.

      Example of condition

      WHEN avg() OF query(A, 15m, now) IS ABOVE 90

       Comments to the example

      The condition will be met if the average value of data from queue A has exceeded 90 in the last 15 minutes.

      To set an additional condition, press . To add a condition through the logical "AND" specify the operand AND at the beginning of the condition. To add a condition through the logical "OR" specify the operand OR at the beginning of the condition.

      Example of condition

      WHEN avg() OF query(A, 15m, now) IS ABOVE 90
      AND max() OF query(A, 15m, now) IS ABOVE 95

       Comments to the example

      The condition will be met if for the last 15 minutes, the average value of data from queue A has exceeded 90 and the maximum has exceeded 95.

      Note

      Additional conditions are checked sequentially in the order they are entered.
   5. If no data or all values are null — notification status if data are missing or have the null value. Possible values: Alerting, No Data, Keep Last State, OK;
   6. If execution error or timeout — сnotification status in case of an error during its execution. Possible values: Alerting, Keep Last State.
4. Select the notification channels in the Send to field.
5. Enter the notification text in the Message field.
6. Save the settings: → enter the comment → Save.