CINQby Quinten Stekelenburg
We all have to deal with it, static, boring email alert notifications. This while the content is about business-critical processes or applications. Unfortunately, this also happens with most of Splunk's alert emails.
The following approach should be familiar, if you have ever created alerts in Splunk. It starts with a need in the organization. Someone wants to be alerted when there’s a problem regarding availability, performance, or on the increase or decrease of a certain figure.
As a Splunk Consultant you will work with the customer demand and translate this need into a concrete solution. You create a search that provides the right data and save that as an alert in Splunk. In the alert settings you set your trigger conditions, and make sure that the alert is send by email to your recipients. In this email you put the required values from your Splunk search through tokens and that's it.
The result is like below, valuable information in a boring email with a piece of text. That being said, there are some good standard options to put the results in the attachments, or to display them as a table and/or add a link to the results. Still, this had to be done differently and is also very easy to achieve.
To stop boring alert emails like above, we are going to make use of HTML and CSS email templates. And that’ what this blog is all about. This blog is divided into two parts. The first part focusses on how to customize Splunk so that HTML is supported. The second part will be an example of how to use and create HTML e-mails with a working example.
When a Splunk environment sends out an email, a Python script called sendemail.py is used in the background. The script itself resides in the location Splunk/etc/apps/search/bin/sendemail.py. To support HTML, we’re going to modify this script slightly so that html is allowed in the body section of the emails.
For the actual modification of the script there are 2 options:
In this blog, we’ll go for the second option. This is also the most likely option, since this change will not affect the rest of the Splunk environment, and will just be applied and work for 1 app. Hence, if you want to go with option 1, you would only have to alter the script (see below second bulletpoint) and save the changes to the Splunk environment.
With the myApp in place we will create an alert as we normally would. So we begin with running a search that would trigger an alert.
After running the above search, we will click “Save As”, “Alert”.
Next we fill out the form as shown below.
What we have now created is an alert that runs via CRON schedule once a minute, and triggers when sendalert=1 is set. Once triggered it sends an email to tom.de.bruijn@cinqict.nl. Then as a subject the field HEADER is used by calling $result.fieldname$. In the message section several fields are used to compose the email body.
1) Now that the alert is active, we can start testing the alerting, we will go back to the search and change sendalert=0 to sendalert=1, we run the search and afterwards save the alert. Then we wait a minute until the CRON triggers the alert. After an email is received change it back to sendalert=0 and save the alert again, to stop receiving e-mails every minute.
2) We will end up with below a-mail alert, that was also shown at the beginning of this blog.
3) Now let’s go back to the alert search and change the text in the MAIL token to an HTML and CSS styled text.
In this blog I’m using a template I created earlier myself. The actual building and styling of a HTML template I left out of this demo. However, to get started and get some ideas, I looked at existing email templates in my mailbox and looked on the internet for email templates. I also use the company branding to match my own template. I built and test the templates by using Jsfiddle, as I found that easy for debugging. Note that your CSS must be embedded in your html and not be a separate file. Get started with just some simple html and when that works expand. And as a final tip, you could use red/orange/green color indications, or images, via tokens to emphasize the different severities of your alerts.
Once you have tested your HTML template in your browser, we need to change it so it becomes a Splunk string. Therefore, we have to escape the double quotes with a leading backslash and once done put a double quote (not escaped) at the beginning and end of the html. Please note, that all fields like DEPARTMENT, TEST, STEP and LOCATION have been submitted as a token input in the EMAIL string.
I ended up with the following:
We save this alert search once again and upon editing the alert we change the MESSAGE section to $result.MAIL$ only, keep Subject and everything else as is.
We can start testing the alert with changing sendalert=0 to sendalert=1 in the alert search again, and off course save it once more. Once a minute has past, an alert will be sent and the HTML in your search will be transformed to HTML in your email Template.
Enthusiastic? Or still need some extra assistance with this? Please let us know, we are happy to help!