In this blog we will state some of the changes and challenges you might face when you upgrade to Splunk 9.1. After reading this blog and checking the latest release notes you should have a better understanding of the upcoming changes when you upgrade to Splunk 9.1.

Preserve search history across search heads

A well know complaint of any Splunk user is that if the Splunk Search heads are in a cluster, they lose their search history. This is because the history of the users is stored locally per Search Head. Starting from Splunk 9.1 the search history is replicated to the other cluster nodes by using a kvstore. This was one highly upvoted Splunk Ideas item and its great to see that Splunk listened to the feedback and implemented it.

For critical Splunk environments the ability to renew the TLS certificate for outputs/server or web without having to restart splunkd is a very welcome change. This helps administrators to minimize downtime of their Splunk environment. 

Starting with Splunk 9.1 it is possible to fill the Dashboards Trusted Domains List with domains/urls that provide content for their dashboards. This will prevent warnings for your user that external content is loaded when they open the dashboard. A other helpful feature is that if you link to external website like your ticket management system you no longer have to click allow if the url is added to the Dashboards Trusted Domains List.

There are a number of improvements to the dashboard studio dashboards in Splunk. 

Export the data results of any visualization to a CSV

If a search does not provide any results you can now hide the panel when you are using a absolute layout dashboard. This is not the exact same functionality as the depends option in simple xml dashboard but this does help. 

You can now export each result to a csv, instead of opening each search and having to export the data there.

Instead of having the inputs in the top left corner it is now possible to place them closer to the panel they affect. This makes it easier for user to “get” how the dashboard is impacted by changes inputs.

After reading the description we tried to find more documentation and ask on slack what the impact of this change will have on searches. The answer from Splunk is as follows:

stats v2 is an "under-the-covers" change that we're telling customers about just as an fyi; it's not anything for you to be concerned about.

You have to specifically edit conf files to have stayed in stats v1 so realistically almost all customers are already using stats v2.

If your organization has a Splunk unlimited license you can now create a high availability cluster of license managers behind a loadbalancer. Unfortunately there is no mention of users that do not have an unlimited license, you could still build a solution yourself but Splunk does not support it out of the box.

When installing the Splunk Universal Forwarder 9.1.x on a fresh machine using the RPM/DEB installer it now creates a splunkfwd account instead of a splunk account. This could cause some problems if you added the Splunk user to certain groups that have access to log files. The new splunkfwd will not have the same rights and you will might not see all the logs that you would expect.  If you are upgrading from an older version you might want to change the default user so that you have 1 standard user across your environment. Admin’s should really check the impact/test the upgrade to make sure you do not run into any issues when upgrading your production environment.  

https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes

https://docs.splunk.com/Documentation/Splunk/latest/DashStudio/WhatNew

In this blog we will state some of the changes and challenges you might face when you upgrade to Splunk 9.1. After reading this blog and checking the latest release notes you should have a better understanding of the upcoming changes when you upgrade to Splunk 9.1

Read this before upgrading to Splunk 9.1

Lately, a lot of people have been asking me about how to deploy web apps using Docker, Inc for their hobby projects. So to help people I threw this quick tutorial about how to deploy an R Shiny web app on Render using their free tier. 


In this tutorial, we are creating a simple R Shiny web app, running it locally, hosting it locally via Docker (desktop), and deploying via Docker and render.com. Sounds fun, right? Let's have a look! 

Deploying R Shiny App via Render Tutorial

Clone this repo
git clone git@github.com:jameshtwose/rshiny-docker-render.git

Open the app in your browser
http://localhost:5000

Build the Docker image
docker build -t rshiny-docker-render .

Run the Docker image
docker run -d -p 5000:5000 rshiny-docker-render

The GitHub repo can be found here: https://github.com/jameshtwose/rshiny-docker-render
And the live demo can be found here: https://rshiny-docker-render.onrender.com/ 

This video tuturial shows you how to deploy an R Shiny webapp, using Docker on Render's free tier.

How to deploy a webapp with Docker on Render

How to deploy an R Shiny webapp with Docker on Render

Our clients are often asking us what is the best technology stack/ architecture to use for their businesses. So today one of our Data Engineers, James Twose, will take us through a comparison of a paid technology stack and an open source technology stack to create, deploy and maintain an MLOps solution. Specifically, a web application that attempts to predict the amount of time the trains in the Netherlands will be disrupted (in minutes) based on the weather conditions on the day (temperature, rain amount).

The following is a presentation of the technology stack comparisons, their learning curves, their pro’s and con’s, and their corresponding prices:

Paid vs open source MLops solution: Introduction

As mentioned in the previous video, here is the setup and demo of the open source, Kubernetes, version of the project:

Paid vs open source MLops solution: Open source version with Kubernetes

And finally, the paid, Databricks, version of the MLOps solution:

Paid vs open source MLops solution: Paid version with Databricks

We hope you found this post enlightening and that what you learned helps you to make decisions for your company, and provided some insights into the process of using these different approaches.  

The GitHub repository for this project, including the presentation, can be found here - https://github.com/cinqict/paid-vs-opensource.

Our clients are often asking us what is the best technology stack/ architecture to use for their businesses. In this video tutorial one of our Data Engineers, James Twose, will take us through a comparison of a paid technology stack and an open source technology stack to create, deploy and maintain an MLOps solution

Paid vs open-source MLops solution

Comparing a paid vs open-source MLops solution

CINQ ICT lanceert deze maand een nieuwe dienst: de KubeScan. Dat doet de Kubernetes Certified Service Provider op KubeCon in de RAI Amsterdam. De KubeScan is specifiek ontwikkeld voor bedrijven die de eerste stap willen zetten naar containerisatie met Kubernetes.

Veel organisaties moderniseren hun IT-landschap door naar de cloud te migreren en door hun softwareproducten in containers te verpakken. De vervolgstap is container-orkestratie door Kubernetes. Dit complexe proces vereist specifieke expertise. Voor bedrijven die op zoek zijn naar schaalbaarheid en flexibiliteit voor hun digitale producten, is de KubeScan van CINQ ICT de oplossing.

“Sommige bedrijven hebben al ideeën over hoe ze met Kubernetes willen werken en vragen om een second opinion, terwijl anderen nog moeten starten met een plan”, vertelt Stefan Koppelaar, commercieel directeur bij CINQ ICT. “De KubeScan biedt in beide gevallen een uitgebreid advies en plan van aanpak. Eventueel gevolgd door de implementatie van de oplossing en training in Kubernetes. Zo beginnen bedrijven met een betrouwbaar en kostenefficiënt fundament aan hun migratie naar de cloud.”

CINQ ICT heeft de KubeScan al bij diverse bedrijven mogen uitvoeren. Zo ontdekten een uitgeverij, een bedrijf dat payroll software maakt, en een softwarebedrijf voor banken en verzekeraars, al het potentieel van Kubernetes voor hun organisatie. 

“Bedrijven die software ontwikkelen, zijn voortdurend op zoek naar manieren om de werking hiervan te verbeteren”, vervolgt Koppelaar. “Ook omdat ze te maken hebben met verschillende IT-omgevingen bij hun klanten. Kubernetes is een populaire oplossing om op een gestandaardiseerde manier software sneller naar een werkende productieomgeving te brengen.”

CINQ ICT is van 19 tot en met 21 april te vinden op KubeCon + CloudNativeCon Europe 2023 bij stand S76.  

CINQ ICT lanceert deze maand zijn KubeScan tijdens KubeCon in de RAI Amsterdam. De KubeScan is specifiek ontwikkeld voor bedrijven die de eerste stap willen zetten naar containerisatie met Kubernetes. Lees hier meer.

CINQ ICT lanceert KubeScan op KubeCon Amsterdam

After a few years as a Splunk Administrator it was finally time for me to get my Splunk Enterprise Architect certification. In this blog I will share my journey through the different courses and what I did to prepare myself for the practical and theoretical exams.

Before starting your Splunk architect courses you should have the following certifications:

Make sure these certifications are still valid since they expire  3 years after obtaining them. You can still follow the courses for Splunk Enterprise Architect if they have expired, but you will not be able to take the last exam at Pearson VUE until you renew them both.

These are the three courses you have to follow:

Architecting Splunk Enterprise Deployments 

During two of the three required courses you will have access to lab environments, use the time you have left after completing the assignment to look at the assignments and make notes on the commands you used.

Make sure you make notes during the courses about best practices and things like search factor/replication factor and site settings, these notes will help you out when you take the practical lab.

Since you are required to do a practical lab to obtain the Enterprise Architect certification I highly recommend building your own Splunk environment lab. If you have a pc or laptop with 16 GB you could use a couple of VM’s with minimal cpu/memory (1/1GB) to build your own Splunk clusters.  For a cluster you could build something like the following set up:

A virtual machine for the license manager, manager node and deployment server role.

2 virtual machines for an indexer cluster

A single search or if you really have some resources left you could make 3 virtual machines as a search head cluster and use the license/manager node as a deployer.

For a license you could request a developer license at:

https://dev.splunk.com/enterprise/dev_license/

That way you really get some experience with setting up a Splunk environment.

For the practical lab you have a Webex call the first 4 hours with one of the Splunk trainers.

Use these 4 hours to make sure that you understand what you should be building. Don’t start at the beginning without looking at all the steps that you should be going through.

Once you have all the information you need to build a Splunk environment you can start but make sure you take notes during the configuration.

You will get 24 hours to complete the environment and sent a mail to the trainer, if you have built Splunk environments in the past then you could finish in 4 to 6 hours. If you have never build a Splunk environment then it will likely take about 10 hours so make sure you have enough time allocated in your calendar to focus on the practical lab.

Once you have passed the practical lab you will get access to the exam at Pearson VUE if you have the power and admin certification, this one is just like any other Splunk exam. Make sure you read through all the provided documents and your notes from the courses  and you should be able to pass this exam and become an official Splunk Enterprise Architect.

https://www.splunk.com/en_us/pdfs/training/splunk-enterprise-certified-architect-track.pdf

Prepare yourself for your Splunk Enterprise Certified Architect exam with tips and tricks from our Splunk consultant.

How to prepare for the Splunk Enterprise Certified Architect exam

In this blog we will state some of the changes and challenges you might face when you upgrade from Splunk 8.2x to Splunk 9.0.4. After reading this blog and checking the latest release notes you should have a better understanding of the upcoming changes when you upgrade to Splunk 9.

Starting with Splunk 9 the classic dashboards should now use a version number, if you do not have a version number the user will see the warning below:

All dashboards that contain the <dashboard> tag will automatically receive the version when upgrading to version 9.0.4. If you upgrade to a lower version you have to add the version=”1.1” yourself. Any dashboards that contain dropdowns will contain the <form> tag instead of the <dashboard> tag. These will not get the version=”1.1” and you have to manually add the version to no longer show the warning.

We recommend to use 9.0.4.1 since 9.0.4 contains a known issue for dashboards that contain <set tokens (found this information on the Splunk Slack):

And this means that the token will no longer work and the dashboard functionality that depends on that token breaks.

To find out if you have any dashboards that might be affected you should run this search that was shared by Becky Burwell on the Splunk Slack:

| rest /servicesNS/-/-/data/ui/views splunk_server=local 
| rename eai:data as DASHBOARD 
| fields id author Type title eai:acl.app DASHBOARD 
| rename author as Owner, title as Name 
| fillnull value=NONE Owner Type AppName Name, DASHBOARD, 
| search  Owner!=nobody  
| xpath field=DASHBOARD outfield=token_name "//condition//set[not(text())]/@token" 
| where mvcount(token_name) > 0 
| mvexpand token_name 
| eventstats dc(Name) as num_dashboards
| table num_dashboards, Owner Name, token_name, DASHBOARD

The workaround is to manually change these dashboards before upgrading to Splunk 9.0.4 by adding version=”1.1” into the <dashboard or <form tag. 

It was stated that this is fixed in version 9.0.4.1 that was released on the 17th of March.

Please note that any dashboards in etc/apps and etc/users will be changed by upgrading, any dashboards that are pushed by deployers in etc/shcluster/apps will not be changed. So pushing apps with dashboards in the default folder without version=”1.1” might undo the changes done on the search heads and show the warning again for users.

With Splunk 9 Dashboard studio now supports the usage of tokens, this is one of the most requested feature for Dashboard studio dashboards. You can now set default tokens or pass tokens when linking to other dashboards , this opens the linked dashboard with the token values so you can see the right information without having to select the options again.

To read all about these changes please see the links at the bottom.

There is a new option to help admin troubleshoot, there is a new index called _configtracker. This index will contain changes to .conf files and their key value pairs. This makes seeing what was changed and when it was changed so much easier without using 3rd party tools.

If you receive ip information in you Splunk environment you can enrich that ip information with the iplocation command. This adds the following fields by default to the results:

City, Country, Region, _time, lat, and lon

Splunk ships its software with a Maxmind database containing this information. It normally gets updated when you upgrade to a new Splunk version. For some use-cases the data needs to be refreshed weekly to prevent the use of inaccurate data and false positive hits when checking for certain countries. To make it easier to update this information you can now update this using the GUI or by placing the file on your search heads and specifying the path in the limits.conf file. For exact instructions please visit the link at the bottom of the page.

Splunk has changed some of the names used in folders and configuration files. An important change is when you have an indexer cluster and you are pushing your apps from the master-apps folder.
 With Splunk 9 you should start using manager-apps instead of master-apps since that will be removed in a feature release. In order to use manager-apps you should copy over all files and folders from the master-apps and remove them in that location. When you start copying you also have to check your conf files for hardcoded paths, the indexer path will change from slave-apps to peer-apps, any hardcoded references to slave-apps will break. Usually paths are used for SSL configuration and you do not want that to break after your upgrade.

There are more changes in Splunk 9.0.x, please read the release notes and check out the Splunk slack channel Splunk_9_upgrade_issues to see what other users are encountering or ask for help.

Splunk Enterprise Release notes: https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes

Splunk slack: https://splunk-usergroups.slack.com

Dashboard studio changes: https://docs.splunk.com/Documentation/Splunk/latest/DashStudio/WhatNew

Configuration change tracker: https://docs.splunk.com/Documentation/Splunk/9.0.4/Troubleshooting/WhatSplunklogsaboutitself#Configuration_Change_Tracker

iplocation command: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Iplocation

Manager-apps: https://docs.splunk.com/Documentation/Splunk/9.0.4/Indexer/Updatepeerconfigurations#Which_directory_to_use:_manager-apps_or_master-apps.3F

Upgrade to Splunk 9 and you will find some new and improved features. Check them out here.

What's new in Splunk 9

Jij bepaalt de richting, wij helpen je om je ambities waar te maken. In deze video vertelt Teun Roefs hoe we er samen voor hebben gezorgd dat hij helemaal op z’n plek zit, zowel bij CINQ als bij de klant. 😄

Werken bij CINQ - De ideale balans

Wil je een andere opdracht, een andere rol of misschien meer investeren in je eigen ontwikkeling? Bij CINQ denken we altijd met je mee.

Enthousiast geworden? We zoeken nog collega’s voor de Data unit. Neem gerust contact met ons op voor een kennismaking 😊

We all have to deal with it, static, boring email alert notifications. This while the content is about business-critical processes or applications. Unfortunately, this also happens with most of Splunk's alert emails.

The following approach should be familiar, if you have ever created alerts in Splunk. It starts with a need in the organization. Someone wants to be alerted when there’s a problem regarding availability, performance, or on the increase or decrease of a certain figure. 

As a Splunk Consultant you will work with the customer demand and translate this need into a concrete solution. You create a search that provides the right data and save that as an alert in Splunk. In the alert settings you set your trigger conditions, and make sure that the alert is send by email to your recipients. In this email you put the required values from your Splunk search through tokens and that's it. 

The result is like below, valuable information in a boring email with a piece of text. That being said, there are some good standard options to put the results in the attachments, or to display them as a table and/or add a link to the results. Still, this had to be done differently and is also very easy to achieve.

To stop boring alert emails like above, we are going to make use of HTML and CSS email templates. And that’ what this blog is all about. This blog is divided into two parts. The first part focusses on how to customize Splunk so that HTML is supported. The second part will be an example of how to use and create HTML e-mails with a working example.

Modify Splunk to support HTML in email alerts.

When a Splunk environment sends out an email, a Python script called sendemail.py is used in the background. The script itself resides in the location Splunk/etc/apps/search/bin/sendemail.py. To support HTML, we’re going to modify this script slightly so that html is allowed in the body section of the emails.

For the actual modification of the script there are 2 options:

Option 1, modify the existing script in the existing place. Then the change is implemented for the entire Splunk environment, and all e-mails will support HTML. 

Option 2 copy the python script to your own custom app, then the HTML email templates are only implemented and supported in this app.

In this blog, we’ll go for the second option. This is also the most likely option, since this change will not affect the rest of the Splunk environment, and will just be applied and work for 1 app. Hence, if you want to go with option 1, you would only have to alter the script (see below second bulletpoint) and save the changes to the Splunk environment.

Copy the existing python script Splunk/etc/apps/search/bin/sendemail.py to your own splunk app (in this tutorial named myApp). Save it in the same app/folder location and rename the script (my_sendmail_html.py). If done correctly it should be in the following location: Splunk/etc/apps/myApp/bin/my_sendmail_html.py

Open the python script (original one for option 1, mysendemail.py for option 2) with a text/code editor and look for the text ${msg|h}. remove the |h so it becomes just ${msg} and save the script again.

Create a commands.conf in the local folder of the myApp app, which assigns the command to the mysendemail.py script just created.
❖   Filename and location: 
▪︎  Splunk/etc/apps/myApp/local/commands.conf
❖   contents commands.conf:


[sendemail]
filename = my_sendmail_html.py
streaming = false
run_in_preview = false
passauth = true
required_fields =
changes_colorder = false
supports_rawargs = true
undo_scheduler_escaping = true
is_risky = true
local = true

Upload the myApp app to the Splunk environment. From now on. HTML is supported in email alerts sent from this app. If you do not apply html in the alerts, the email will be sent as usual.

With the myApp in place we will create an alert as we normally would. So we begin with running a search that would trigger an alert.

| makeresults
``` send alert only if set to 1, this so we have a trigger mechanism ```
| eval sendalert=0
``` alert header will be used as subject ```
| eval HEADER="HTML Splunk Alerting CinqICT."
``` fields that will be used in the email ```
| eval SUMMARY="A Positive Normal Performance alert for department Data Businessunits and test HTML Email Alerting at step Send Fancy Email was triggered."
| eval DEPARTMENT="Data Businessunits"
| eval TEST="HTML Email Alerting"
| eval STEP="Send Fancy Email"
| eval LOCATION="Burgemeester Stramanweg 101, 1101 AA Amsterdam."

After running the above search, we will click “Save As”, “Alert”.

Next we fill out the form as shown below.

What we have now created is an alert that runs via CRON schedule once a minute, and triggers when sendalert=1 is set. Once triggered it sends an email to tom.de.bruijn@cinqict.nl.  Then as a subject the field HEADER is used by calling $result.fieldname$. In the message section several fields are used to compose the email body.

1)    Now that the alert is active, we can start testing the alerting, we will go back to the search and change sendalert=0 to sendalert=1, we run the search and afterwards save the alert. Then we wait a minute until the CRON triggers the alert. After an email is received change it back to sendalert=0 and save the alert again, to stop receiving e-mails every minute.

2)   We will end up with below a-mail alert, that was also shown at the beginning of this blog.

3) Now let’s go back to the alert search and change the text in the MAIL token to an HTML and CSS styled text.

In this blog I’m using a template I created earlier myself. The actual building and styling of a HTML template I left out of this demo. However, to get started and get some ideas, I looked at existing email templates in my mailbox and looked on the internet for email templates. I also use the company branding to match my own template. I built and test the templates by using Jsfiddle, as I found that easy for debugging. Note that your CSS must be embedded in your html and not be a separate file. Get started with just some simple html and when that works expand. And as a final tip, you could use red/orange/green color indications, or images, via tokens to emphasize the different severities of your alerts.

Once you have tested your HTML template in your browser, we need to change it so it becomes a Splunk string. Therefore, we have to escape the double quotes with a leading backslash and once done put a double quote (not escaped) at the beginning and end of the html. Please note, that all fields like DEPARTMENT, TEST, STEP and LOCATION have been submitted as a token input in the EMAIL string.

| makeresults
``` send alert only if set to 1, this so we have a trigger mechanism ```
| eval sendalert=0
``` alert header will be used as subject ```
| eval HEADER="HTML Splunk Alerting CinqICT."
``` fields that will be used in the email ```
| eval SUMMARY="A Severe Performance alert for department Data Businessunits and test HTML Email Alerting at step Send Fancy Email was triggered."
| eval DEPARTMENT="Data Businessunits"
| eval TEST="HTML Email Alerting"
| eval STEP="Send Fancy Email"
| eval LOCATION="Burgemeester Stramanweg 101, 1101 AA Amsterdam."
``` html email template ```
| eval MAIL="<body style=\"background-color: #f1f1f1; \"><header>   <div class=\"panel-body html\" style=\"max-width:1000px; margin: 10px auto auto auto;\">     <div style=\"min-height:130px; height:auto;border-radius: 5px; text-align: left; background-color: #545454;\">        <img src=\"https://media-exp1.licdn.com/dms/image/C4E0BAQHp5PH21idHPg/company-logo_200_200/0/1652183310444?e=1667433600&v=beta&t=B0VuCkIHaYp2J_YOzNJjbTnZQWnR0qQ9g7vp6w6Pv7w\" width=\"125px\" style=\"padding: 0px 20px 0px 0px;\" align=\"right\">       <table>         <tbody>           <tr>            <td height=\"10%\" width=\"750\" style=\"padding:0cm 0pt 0cm 5.4pt\">            <span style=\"font-size: 35px;font-weight: bold; color: #ffffff; \">".HEADER."</span>            </td>           </tr>           <tr>            <td height=\"1\" width=\"750\" style=\"padding:0cm 0pt 0cm 5.4pt\">            <span style=\"font-size: 16px; color: #ffffff; color: #FBD913;\">".SUMMARY."</span>            </td>           </tr>          </tbody>         </table>         </div></div></header><section><div class=\"panel-body html\" style=\"max-width:1000px; margin: 10px auto auto auto;\">   <div style=\"line-height: 25px; height:25px;border-radius: 5px; text-align: left; background-color: #545454; \">   <span style=\"font-size: 14px; font-weight: bold; color: #FBD913; padding-left: 0.5em; margin: 0; position: relative;\">Details listed</span>   </div></div><div class=\"panel-body html\" style=\"background-color: #DFDFDF;border-radius: 5px; max-width:1000px; margin: 10px auto auto auto;\">     <span style=\"font-size: 13px; color: #000000; position: relative; left: 2%;\">     <table cellspacing=\"0\" cellpadding=\"0\" style=\"border-spacing: 0;border-collapse:collapse;border:none\"> <tbody> <tr>  <td width=\"142\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 0pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Department</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">".DEPARTMENT."</span></b></p>  </td> </tr> <tr>  <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Test</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">".TEST."</span></b></p>  </td> </tr> <tr>  <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Step</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">".STEP."</span></b></p>  </td> </tr> <tr>  <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Event Type</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">Response Time</span></b></p>  </td> </tr> <tr>  <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Location/Nodes</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span lang=\"EN-US\" style=\"font-size:11.5pt;font-family:  &quot;Segoe UI&quot;,sans-serif;color:#201F1E\">".LOCATION."</span></b></p>  </td> </tr> <tr>  <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Responsetime Message</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span lang=\"EN-US\" style=\"font-size:11.5pt;font-family:  &quot;Segoe UI&quot;,sans-serif;color:#201F1E\">Performance restored from warning to  normal threshold!</span></b></p>  </td> </tr> <tr>  <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Thresholds</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">Warning = 5, High = 8</span></b></p>  </td> </tr> <tr>  <td width=\"142\" valign=\"top\" style=\"width:106.35pt;border:none;border-right:solid windowtext 1.0pt;  padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><span style=\"font-size:11.5pt;font-family:&quot;Segoe UI&quot;,sans-serif;  color:#201F1E\">• Alert Time</span></p>  </td>  <td width=\"434\" valign=\"top\" style=\"width:325.4pt;border:none;padding:0cm 5.4pt 0cm 5.4pt\">  <p class=\"MsoNormal\"><b><span lang=\"EN-US\" style=\"font-size:11.5pt;font-family:  &quot;Segoe UI&quot;,sans-serif;color:#201F1E\">Alert Start Time: 2022-07-25 11:44:44<br>  Alert End Time: 2022-07-25 12:59:00<br>  Total Alert Time: 1 hour, 14 minutes and 16 seconds.</span></b></p>  </td> </tr></tbody></table></span>   </div></section><footer><div class=\"panel-body html\" style=\"max-width:1000px; margin: 10px auto auto auto;\">   <div style=\"line-height: 25px; height:25px;border-radius: 5px; text-align: left; background-color: #545454; \">   <span style=\"font-size: 14px; font-weight: bold; color: #FBD913; padding-left: 0.5em; margin: 0; position: relative;\">Extra Information</span>   </div></div>   <div class=\"panel-body html\" style=\"background-color: #DFDFDF; height: auto; border-radius: 5px; max-width:1000px; margin: 5px auto auto auto;\">   <a href=\"https://www.linkedin.com/in/tcdebruijn/\">     <span style=\"font-size: 13px; color: #000000; position: relative; left: 2%;\">Confluence Splunk Dashboards</span>   </a>   </div>        <div class=\"panel-body html\" style=\"background-color: #DFDFDF; height: auto; border-radius: 5px; max-width:1000px; margin: 5px auto auto auto;\">   <a href=\"https://www.linkedin.com/in/tcdebruijn/\">     <span style=\"font-size: 13px; color: #000000; position: relative; left: 2%;\">Jira Servicedesk</span>   </a>   </div></footer></body>"

We save this alert search once again and upon editing the alert we change the MESSAGE section to $result.MAIL$ only, keep Subject and everything else as is.

We can start testing the alert with changing sendalert=0 to sendalert=1 in the alert search again, and off course save it once more. Once a minute has past, an alert will be sent and the HTML in your search will be transformed to HTML in your email Template.

Enthusiastic? Or still need some extra assistance with this? Please let us know, we are happy to help!

Stop boring email alerts!

In this blog I would like to talk about Cribl and what its possibilities are and what their certification track currently looks like. 

After seeing a demo of the possibilities of Cribl I wanted to learn more about the product and how we could use it. I have been a Splunk administrator for a few years now and while following the demo I could see how Cribl solves some of the issues and struggles I face as a Splunk administrator. 

Cribl positions itself as a layer between your data sources and your data analytics tools like Splunk, Elastic, Databricks & more.

They call this layer: Cribl Stream. It can receive data and then change, transform and reduce the data before sending it off to different destinations. Below are a few of the options that Stream offers:

Change the data format of the data going out, for example you could change it to JSON.

Drop all the empty/null value fields in the data.

Sent a portion of the event to your SIEM and store the raw log event into a S3 bucket as an archive.

Replay that archive S3 bucket to load the full events into your SIEM when you need to investigate/troubleshoot an issue.

Mask sensitive information for 1 destination but send all data to a 2nd data analytics tool.

Receive data from existing agents as the Splunk forwarder or Elastic Beats.

All of these options are available in a GUI and you can see the effects of changes you are making before actually applying them. After learning about the possibilities I decided it was worth to do a proof of concept in my local test environment.

The goal of the proof of concept was to see how I could send data from a network appliance, and get it into Splunk in a JSON format while also reducing the noise in the events.

There are several options for installing Stream. You can install it on your local machine, use Docker containers or you can make use of their free cloud environment to test things. I installed Stream on a Linux virtual machine and after a few minutes it was already up and running.

The first thing I needed to do was to send data to Stream. Since it is able to receive syslog data, I logged into my router/firewall and started to send my network traffic information to Stream using syslog. In the Stream GUI I started to look at the data coming in, there were a few fields to remove and I wanted to rename some of the other fields. 
After a few minutes I had prepared the changes and could see what would happen if I pushed the change out. The data was almost ready, I only needed to change the format to JSON and add an index and sourcetype field since I was going to send the data to my local Splunk HEC endpoint. 
All in all the entire POC cost about an hour to do and I was really happy with the results.

Besides Stream they also offer the edge software, which is an agent you can install on Linux or Windows servers. The main point of the agent is that it can be used to discover data and also route the data to different types of receivers like, Splunk, Elastic or Databricks. This could help to reduce the amount of agents needed on endpoints and have 1 agent to get the data from your endpoints.

To learn more about Stream, I went to https://cribl.io/university/ and saw that they offer the certification tracks for free. Currently there are two certifications that you can acquire:

For anyone wanting to learn more about Stream, I highly recommend to start with Stream User. It is a combination of video’s, questions and their own sandbox environment which lets you make the changes in the actual product. You really get a good sense of the possibilities of Stream and how you should use the product. 

After finishing the Stream User I went along and followed the Stream Admin track, they really go in depth and help you understand what is happening inside of Stream. If you are thinking about doing a Proof of concept in an enterprise environment this will really help you understand how to set it up. 

I hope this blog gives a feel of what Cribl is and does, and if you have any questions please feel free to reach out to me on LinkedIn.

In this blog we introduce you to the Cribl product. What does it do and what does it's certification track look like?

The Cribl product and their certifications

Certificaten, cruciaal om je server verkeer te beveiligen. Maar ze staan bij veel mensen ook bekend als probleem-veroorzakers. Eén van de problemen die we bij onze klanten meemaken is dat de certificaten niet tijdig worden vervangen. Met Splunk hebben we een krachtige tool in handen, die we ook voor dit probleem kunnen inzetten.

Een Splunk installatie heeft een aantal standaard componenten. Zo komt Splunk standaard met zijn eigen Python versie, maar ook OpenSSL is een van deze componenten. Met OpenSSL kunnen we certificaten uitlezen en dat maakt het mogelijk om monitoring in te richten.

Het volgende commando kan de start- en einddatum van een certificaat tonen:

splunk cmd openssl x509 -inform pem -noout -startdate -enddate

Om dit te kunnen doen hebben we eerst het certificaat nodig. Indien je certificaten die op de server staan zou willen monitoren, kan je die waarschijnlijk nog wel vinden. Maar deze oplossing is juist zo krachtig omdat je de geldigheid van certificaten kunt monitoren waar jij zelf gebruik van maakt. Om certificaten te tonen van andere servers of applicaties kunnen we het volgende commando gebruiken:

splunk cmd openssl s_client -showcerts -connect cinqict.nl:443

In dit voorbeeld maakt OpenSSL een verbinding met de website van CINQ over poort 443. Op deze manier kunnen allerlei applicaties op verschillende poorten benaderd worden. Combineer deze commando’s, filter wat op de output en je hebt bijvoorbeeld dit commando:

splunk cmd openssl s_client -showcerts -servername cinqict.nl -connect cinqict.nl:443  2>/dev/null | splunk cmd openssl x509 -inform pem -noout -startdate -enddate

notAfter=Apr 11 07:13:54 2022 GMT
notBefore=Jan 11 07:13:55 2022 GMT

De volgende stap die we nu moeten doen, is zorgen dat we hiervan een scripted input maken in Splunk.

Het commando dat we gemaakt hebben zetten we om in een Linux shell script. We zetten de hostname en de poort om in parameter. Het scriptje noemen we: sslcheck.sh. 

echo hostname=$1 port=$2
echo | splunk cmd openssl s_client -showcerts -servername $1 -connect $1:$2 2>/dev/null | splunk cmd openssl x509 -inform pem -noout -startdate -enddate

Vergeet niet het scriptje executable te maken (chmod +x) en probeer het uit:

De output van dit scriptje ziet er als volgt uit:

hostname=cinqict.nl port=443
notBefore=Jan 11 07:13:55 2022 GMT
notAfter=Apr 11 07:13:54 2022 GMT

Als we de output van dit script in Splunk indexeren, kunnen we proactief gaan monitoren! Daar hebben we een zogeheten scripted input voor nodig. Maak in je Splunk home directory in etc/apps een nieuwe app aan: TA_CertificateMonitoring en maak twee subdirectories: bin en local. In de bin directory plaats je het sslcheck.sh script en in de local directory maak je een inputs.conf aan.

[script:///$SPLUNK_HOME/etc/apps/TA_SSLCertificateCheck/bin/sslcheck.sh {hostname} {port}]
index = {indexname}
sourcetype = sslcheck
disabled = false
interval = 43200

Als laatste moeten we de Sourcetype nog definiëren, alle data in Splunk bevat een tijdswaarde. In de output van het script zit alleen de start- en einddatum van het certificaat, niet het tijdstip van monitoren. Daarom voegen we in de Sourcetype definitie toe dat de huidige systeem-tijd hiervoor gebruikt moet worden. In de local directory van de TA maak je een props.conf aan die er alsvolgt uitziet:

[sslcheck]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
BREAK_ONLY_BEFORE = hostname

Zodra we de scripted input hebben aangemaakt en Splunk herstart hebben, is de input actief. Zoek in de index die je in de inputs.conf hebt opgegeven naar de data:


Er zijn meerdere manieren om de einddatum juist in een veld te krijgen. Je kunt hiervoor bijvoorbeeld gebruik maken van een field extraction of een calculated field, waarbij je een EVAL functie kunt gebruiken. In dit voorbeeld lossen we het voor het gemak even op met een reguliere expressie. Vervolgens zetten we het veld om naar een epoch timestamp zodat we dit kunnen vergelijken met de huidige datum tijd.

index=_internal sourcetype=sslcheck hostname=cinqict.nl
| rex "notAfter=(?<enddate>[^\n]+)"
| eval enddate = strptime(enddate, "%b %d %H:%M:%S %Y %Z”)

Nu kunnen we een alert gaan maken die ons twee maanden van te voren informeert, zodat we tijdig het certificaat kunnen verlengen of vervangen.

index=_internal sourcetype=sslcheck hostname=cinqict.nl
| rex "notAfter=(?<enddate>[^\n]+)"
| eval enddate = strptime(enddate, "%b %d %H:%M:%S %Y %Z")
| where enddate < relative_time(now(), "+2m")

Sla deze search op als alert, laat deze dagelijks draaien en mail bijvoorbeeld naar het gehele team. Misschien is er op jouw Splunk omgeving wel een custom alert action gemaakt, waardoor je hier meteen een actie van kunt maken op je sprint planning!

Mocht je na het lezen van deze blogpost vragen hebben over het monitoren van certificaten, neem dan gerust contact met ons op. We denken graag met je mee!

Meer weten over certificaten monitoren en vervangen?

Luister naar onze podcast-aflevering over het geautomatiseerd vervangen van TLS certificaten. 

Lees de blogpost over het geautomatiseerd vervangen van certificaten met Hashicorp Vault en cert-manager.

Splunk kan ingezet worden om de geldigheid van certificaten op je servers te monitoren. Hoe je dit opzet lees je in dit blog.

Certificaten monitoren met Splunk

Splunk® IT Service Intelligence (ITSI); De oplossing voor monitoring en analyse, welke aangedreven door machine learning en kunstmatige intelligentie inzicht biedt in de gezondheid van kritieke IT- en zakelijke diensten en hun infrastructuur. 

Een uitleg als bovenstaande, gevolgd door de verschillende usecases die Splunk ITSI vervult is wat ik vooral terugvind op het internet. Wat ik nog weinig zie zijn de ervaringen uit de praktijk of technische how-to’s. Zonde, want na een korte kennismaking met ITSI is vaak de reactie: ‘ziet er super gaaf uit’ – kort gevolgd door – ‘maar wat is er nodig voor de inrichting? Hoe werkt het en/of waar moet ik beginnen?’  

In dit artikel geef ik antwoord op deze vragen en neem ik jullie graag mee naar het startpunt van ITSI, namelijk het aansluiten van een databron. Ik ga wat meer in op de techniek achter de oplossing en laat aan de hand van een praktijkvoorbeeld zien hoe je de metrieken van Applicatie Performance Monitoring op Splunk ITSI aansluit met daarbij een integratie naar ServiceNow. 

High Level Design  - APM integratie Splunk ITSI & ServiceNow

Applicatie Performance Monitoring integratie naar Splunk

Als uitgangspunt nemen wij een bestaande databron, de APM-monitoring in een organisatie, links in het HLD als cloud oplossing weergegeven. Dat is de monitoring die door middel van metingen inzicht biedt in de beschikbaarheid- en performance van een applicatie, dienst of een omgeving. Uit de APM-monitoring komen inzichten. Dit zijn de ruwe meetresultaten maar ook al verrijkte data zoals beschikbaarheid- of performance alerts. 

Laten we kort stilstaan bij het versturen van de data naar Splunk. Veelal wordt hiervoor gebruik gemaakt van een webhook. Splunk biedt hiervoor zelf de HTTP Event Collector (HEC) aan. Middels HEC wordt de data op een veilige manier naar de Splunk omgeving verstuurd waarnaar het direct in de juiste index wordt opgeslagen. In het HLD noemen wij deze index de APM_Name_Private_index. 

Met de alert-data in Splunk is de vervolgstap deze gereed te maken voor Splunk ITSI. Hiervoor is het nodig om de structuur van de data aan te passen naar een structuur die geschikt is voor ITSI. Dit aanpassen van de data naar een geschikt formaat wordt normaliseren genoemd en licht ik graag even toe.

Normaliseren is nodig zodat Splunk ITSI begrijpt wat voor data het binnen krijgt en wat het vervolgens met een event moet doen. Normaliseren lijkt in het begin wat omslachtig, de data zit immers al in Splunk. Het is daarom waarschijnlijk al eerder genormaliseerd tijdens het indexeren, dus waarom moet je het nog een keer aanpassen?

Zie het als volgt; wanneer je meerdere databronnen aansluit op Splunk, dan is de data los van elkaar goed te begrijpen en ook toepasbaar voor ieders doeleinde. Het is echter moeilijk om de data voor hetzelfde doeleinde te gebruiken. Wanneer de data genormaliseerd wordt naar een eenduidig bestandsformaat met daarbij een eenduidige key-value indeling is het voor Splunk ITSI mogelijk wat met de events te kunnen doen. 

Belangrijk hierbij is dus dat de key-value indeling door normalisatie hetzelfde wordt. Als voorbeeld: waar in de APM Name Private index links in de afbeelding weergegeven, een veld als responstijd wordt aangeduid met de key-value combinatie performance=value, zal in index 2 dat veld bijvoorbeeld timer=value zijn en in index 3 responsetime=value. Door normalisatie kan dat voor alle drie de databronnen eenduidig aangepast worden naar itsi_duration=value. 

Voor het normaliseren van data naar ITSI-events zijn regels en richtlijnen opgesteld, welke door Splunk zijn opgenomen in het Common Interface Model (CIM). Hierin staat beschreven welk veld(key) welk doeleinde(value) heeft en worden er voorbeelden gegeven voor hoe deze toegepast kunnen worden. Los van het CIM zijn er ook Cheat Sheets en Templates beschikbaar gesteld voor veel gebruikte integraties.

Data Normalisatie middels een scheduled search

Het normaliseren – oftewel het hernoemen van velden – gaat in de praktijk middels een Scheduled Search. Zoals de naam al zegt is een Scheduled Search een zoekopdracht in Splunk die je herhaald kan laten uitvoeren. Voor dit doeleinde kijkt de zoekopdracht iedere minuut of er in de APM_Name_Private_index nieuwe events binnen zijn gekomen. Indien dit het geval is worden de velden uit de events middels meerdere eval commando’s aangepast naar ITSI-velden en middels een collect commando weggeschreven naar een aparte summary index, in het HLD genaamd apmname_alerts_itsi_public. 

Onderstaand een stuk voorbeeld code SPL waarbij de eval commando’s voor de belangrijkste ITSI-velden worden toegelicht. In de praktijk dient deze code aangepast te worden op de eigen situatie. 

Splunk consultant Tom de Bruijn  legt in 5 stappen uit uit hoe je Applicatie Performance Monitoring kan aansluiten op Splunk IT Service Intelligence.

Onze laatste blogs over Data