Geschreven door Tom de Bruijn
Stop boring email alerts!
We all have to deal with it, static, boring email alert notifications. This while the content is about business-critical processes or applications. Unfortunately, this also happens with most of Splunk's alert emails.
The following approach should be familiar, if you have ever created alerts in Splunk. It starts with a need in the organization. Someone wants to be alerted when there’s a problem regarding availability, performance, or on the increase or decrease of a certain figure.
As a Splunk Consultant you will work with the customer demand and translate this need into a concrete solution. You create a search that provides the right data and save that as an alert in Splunk. In the alert settings you set your trigger conditions, and make sure that the alert is send by email to your recipients. In this email you put the required values from your Splunk search through tokens and that's it.
The result is like below, valuable information in a boring email with a piece of text. That being said, there are some good standard options to put the results in the attachments, or to display them as a table and/or add a link to the results. Still, this had to be done differently and is also very easy to achieve.
To stop boring alert emails like above, we are going to make use of HTML and CSS email templates. And that’ what this blog is all about. This blog is divided into two parts. The first part focusses on how to customize Splunk so that HTML is supported. The second part will be an example of how to use and create HTML e-mails with a working example.
Modify Splunk to support HTML in email alerts.
When a Splunk environment sends out an email, a Python script called sendemail.py is used in the background. The script itself resides in the location Splunk/etc/apps/search/bin/sendemail.py. To support HTML, we’re going to modify this script slightly so that html is allowed in the body section of the emails.
For the actual modification of the script there are 2 options:
- Option 1, modify the existing script in the existing place. Then the change is implemented for the entire Splunk environment, and all e-mails will support HTML.
- Option 2 copy the python script to your own custom app, then the HTML email templates are only implemented and supported in this app.
In this blog, we’ll go for the second option. This is also the most likely option, since this change will not affect the rest of the Splunk environment, and will just be applied and work for 1 app. Hence, if you want to go with option 1, you would only have to alter the script (see below second bulletpoint) and save the changes to the Splunk environment.
- Copy the existing python script Splunk/etc/apps/search/bin/sendemail.py to your own splunk app (in this tutorial named myApp). Save it in the same app/folder location and rename the script (my_sendmail_html.py). If done correctly it should be in the following location: Splunk/etc/apps/myApp/bin/my_sendmail_html.py
- Open the python script (original one for option 1, mysendemail.py for option 2) with a text/code editor and look for the text ${msg|h}. remove the |h so it becomes just ${msg} and save the script again.
- Create a commands.conf in the local folder of the myApp app, which assigns the command to the mysendemail.py script just created.
❖ Filename and location:
▪︎ Splunk/etc/apps/myApp/local/commands.conf
❖ contents commands.conf:
[sendemail]
filename = my_sendmail_html.py
streaming = false
run_in_preview = false
passauth = true
required_fields =
changes_colorder = false
supports_rawargs = true
undo_scheduler_escaping = true
is_risky = true
local = true- Upload the myApp app to the Splunk environment. From now on. HTML is supported in email alerts sent from this app. If you do not apply html in the alerts, the email will be sent as usual.
Create an HTML Alert:
With the myApp in place we will create an alert as we normally would. So we begin with running a search that would trigger an alert.
| makeresults
``` send alert only if set to 1, this so we have a trigger mechanism ```
| eval sendalert=0
``` alert header will be used as subject ```
| eval HEADER="HTML Splunk Alerting CinqICT."
``` fields that will be used in the email ```
| eval SUMMARY="A Positive Normal Performance alert for department Data Businessunits and test HTML Email Alerting at step Send Fancy Email was triggered."
| eval DEPARTMENT="Data Businessunits"
| eval TEST="HTML Email Alerting"
| eval STEP="Send Fancy Email"
| eval LOCATION="Burgemeester Stramanweg 101, 1101 AA Amsterdam."After running the above search, we will click “Save As”, “Alert”.
Next we fill out the form as shown below.
What we have now created is an alert that runs via CRON schedule once a minute, and triggers when sendalert=1 is set. Once triggered it sends an email to tom.de.bruijn@cinqict.nl. Then as a subject the field HEADER is used by calling $result.fieldname$. In the message section several fields are used to compose the email body.
1) Now that the alert is active, we can start testing the alerting, we will go back to the search and change sendalert=0 to sendalert=1, we run the search and afterwards save the alert. Then we wait a minute until the CRON triggers the alert. After an email is received change it back to sendalert=0 and save the alert again, to stop receiving e-mails every minute.
2) We will end up with below a-mail alert, that was also shown at the beginning of this blog.
3) Now let’s go back to the alert search and change the text in the MAIL token to an HTML and CSS styled text.
In this blog I’m using a template I created earlier myself. The actual building and styling of a HTML template I left out of this demo. However, to get started and get some ideas, I looked at existing email templates in my mailbox and looked on the internet for email templates. I also use the company branding to match my own template. I built and test the templates by using Jsfiddle, as I found that easy for debugging. Note that your CSS must be embedded in your html and not be a separate file. Get started with just some simple html and when that works expand. And as a final tip, you could use red/orange/green color indications, or images, via tokens to emphasize the different severities of your alerts.
Once you have tested your HTML template in your browser, we need to change it so it becomes a Splunk string. Therefore, we have to escape the double quotes with a leading backslash and once done put a double quote (not escaped) at the beginning and end of the html. Please note, that all fields like DEPARTMENT, TEST, STEP and LOCATION have been submitted as a token input in the EMAIL string.
I ended up with the following:
| makeresults
``` send alert only if set to 1, this so we have a trigger mechanism ```
| eval sendalert=0
``` alert header will be used as subject ```
| eval HEADER="HTML Splunk Alerting CinqICT."
``` fields that will be used in the email ```
| eval SUMMARY="A Severe Performance alert for department Data Businessunits and test HTML Email Alerting at step Send Fancy Email was triggered."
| eval DEPARTMENT="Data Businessunits"
| eval TEST="HTML Email Alerting"
| eval STEP="Send Fancy Email"
| eval LOCATION="Burgemeester Stramanweg 101, 1101 AA Amsterdam."
``` html email template ```
| eval MAIL="<body style=\"background-color: #f1f1f1; \"><header> <div class=\"panel-body html\" style=\"max-width:1000px; margin: 10px auto auto auto;\"> <div style=\"min-height:130px; height:auto;border-radius: 5px; text-align: left; background-color: #545454;\"> <img src=\"https://media-exp1.licdn.com/dms/image/C4E0BAQHp5PH21idHPg/company-logo_200_200/0/1652183310444?e=1667433600&v=beta&t=B0VuCkIHaYp2J_YOzNJjbTnZQWnR0qQ9g7vp6w6Pv7w\" width=\"125px\" style=\"padding: 0px 20px 0px 0px;\" align=\"right\"> <table> <tbody> <tr> <td height=\"10%\" width=\"750\" style=\"padding:0cm 0pt 0cm 5.4pt\"> <span style=\"font-size: 35px;font-weight: bold; color: #ffffff; \">".HEADER."</span> </td> </tr> <tr> <td height=\"1\" width=\"750\" style=\"padding:0cm 0pt 0cm 5.4pt\"> <span style=\"font-size: 16px; color: #ffffff; color: #FBD913;\">".SUMMARY."</span> </td> </tr> </tbody> </table> </div></div></header><section><div class=\"panel-body html\" style=\"max-width:1000px; margin: 10px auto auto auto;\"> <div style=\"line-height: 25px; height:25px;border-radius: 5px; text-align: left; background-color: #545454; \"> <span style=\"font-size: 14px; font-weight: bold; color: #FBD913; padding-left: 0.5em; margin: 0; position: relative;\">Details listed</span> </div></div><div class=\"panel-body html\" style=\"background-color: #DFDFDF;border-radius: 5px; max-width:1000px; margin: 10px auto auto auto;\"> <span style=\"font-size: 13px; color: #000000; position: relative; left: 2%;\"> <table cellspacing=\"0\" cellpadding=\"0\" style=\"border-spacing: 0;border-collapse:collapse;border:none\"> <tbody> <tr> <td width=\"142\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 0pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Department</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">".DEPARTMENT."</span></b></p> </td> </tr> <tr> <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Test</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">".TEST."</span></b></p> </td> </tr> <tr> <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Step</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">".STEP."</span></b></p> </td> </tr> <tr> <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Event Type</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">Response Time</span></b></p> </td> </tr> <tr> <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Location/Nodes</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span lang=\"EN-US\" style=\"font-size:11.5pt;font-family: "Segoe UI",sans-serif;color:#201F1E\">".LOCATION."</span></b></p> </td> </tr> <tr> <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Responsetime Message</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span lang=\"EN-US\" style=\"font-size:11.5pt;font-family: "Segoe UI",sans-serif;color:#201F1E\">Performance restored from warning to normal threshold!</span></b></p> </td> </tr> <tr> <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Thresholds</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">Warning = 5, High = 8</span></b></p> </td> </tr> <tr> <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt; padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:"Segoe UI",sans-serif; color:#201F1E\">• Alert Time</span></p> </td> <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\"> <p class=\"MsoNormal\"><b><span lang=\"EN-US\" style=\"font-size:11.5pt;font-family: "Segoe UI",sans-serif;color:#201F1E\">Alert Start Time: 2022-07-25 11:44:44<br> Alert End Time: 2022-07-25 12:59:00<br> Total Alert Time: 1 hour, 14 minutes and 16 seconds.</span></b></p> </td> </tr></tbody></table></span> </div></section><footer><div class=\"panel-body html\" style=\"max-width:1000px; margin: 10px auto auto auto;\"> <div style=\"line-height: 25px; height:25px;border-radius: 5px; text-align: left; background-color: #545454; \"> <span style=\"font-size: 14px; font-weight: bold; color: #FBD913; padding-left: 0.5em; margin: 0; position: relative;\">Extra Information</span> </div></div> <div class=\"panel-body html\" style=\"background-color: #DFDFDF; height: auto; border-radius: 5px; max-width:1000px; margin: 5px auto auto auto;\"> <a href=\"https://www.linkedin.com/in/tcdebruijn/\"> <span style=\"font-size: 13px; color: #000000; position: relative; left: 2%;\">Confluence Splunk Dashboards</span> </a> </div> <div class=\"panel-body html\" style=\"background-color: #DFDFDF; height: auto; border-radius: 5px; max-width:1000px; margin: 5px auto auto auto;\"> <a href=\"https://www.linkedin.com/in/tcdebruijn/\"> <span style=\"font-size: 13px; color: #000000; position: relative; left: 2%;\">Jira Servicedesk</span> </a> </div></footer></body>"We save this alert search once again and upon editing the alert we change the MESSAGE section to $result.MAIL$ only, keep Subject and everything else as is.
We can start testing the alert with changing sendalert=0 to sendalert=1 in the alert search again, and off course save it once more. Once a minute has past, an alert will be sent and the HTML in your search will be transformed to HTML in your email Template.
Enthusiastic? Or still need some extra assistance with this? Please let us know, we are happy to help!