Read this before upgrading to Splunk 9.1

In this blog we will state some of the changes and challenges you might face when you upgrade to Splunk 9.1. After reading this blog and checking the latest release notes you should have a better understanding of the upcoming changes when you upgrade to Splunk 9.1.

Preserve search history across search heads

A well know complaint of any Splunk user is that if the Splunk Search heads are in a cluster, they lose their search history. This is because the history of the users is stored locally per Search Head. Starting from Splunk 9.1 the search history is replicated to the other cluster nodes by using a kvstore. This was one highly upvoted Splunk Ideas item and its great to see that Splunk listened to the feedback and implemented it.

hot-reload for TLS

For critical Splunk environments the ability to renew the TLS certificate for outputs/server or web without having to restart splunkd is a very welcome change. This helps administrators to minimize downtime of their Splunk environment.

External content Splunk dashboard

Starting with Splunk 9.1 it is possible to fill the Dashboards Trusted Domains List with domains/urls that provide content for their dashboards. This will prevent warnings for your user that external content is loaded when they open the dashboard. A other helpful feature is that if you link to external website like your ticket management system you no longer have to click allow if the url is added to the Dashboards Trusted Domains List.

Dashboard Studio improvements

There are a number of improvements to the dashboard studio dashboards in Splunk.

• Show or hide panels in Absolute layout
• Export the data results of any visualization to a CSV
• Inputs available in the canvas

Show or hide panels in Absolute layout

If a search does not provide any results you can now hide the panel when you are using a absolute layout dashboard. This is not the exact same functionality as the depends option in simple xml dashboard but this does help.

Export the data results of any visualization to a CSV

You can now export each result to a csv, instead of opening each search and having to export the data there.

Inputs available in the canvas

Instead of having the inputs in the top left corner it is now possible to place them closer to the panel they affect. This makes it easier for user to “get” how the dashboard is impacted by changes inputs.

Stats V1 deprecation

After reading the description we tried to find more documentation and ask on slack what the impact of this change will have on searches. The answer from Splunk is as follows:

stats v2 is an "under-the-covers" change that we're telling customers about just as an fyi; it's not anything for you to be concerned about.

You have to specifically edit conf files to have stayed in stats v1 so realistically almost all customers are already using stats v2.

License manager redundancy

If your organization has a Splunk unlimited license you can now create a high availability cluster of license managers behind a loadbalancer. Unfortunately there is no mention of users that do not have an unlimited license, you could still build a solution yourself but Splunk does not support it out of the box.

Splunk Universal Forwarder

When installing the Splunk Universal Forwarder 9.1.x on a fresh machine using the RPM/DEB installer it now creates a splunkfwd account instead of a splunk account. This could cause some problems if you added the Splunk user to certain groups that have access to log files. The new splunkfwd will not have the same rights and you will might not see all the logs that you would expect.  If you are upgrading from an older version you might want to change the default user so that you have 1 standard user across your environment. Admin’s should really check the impact/test the upgrade to make sure you do not run into any issues when upgrading your production environment.  

Links:

Splunk Enterprise Release notes:

https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes

Splunk slack:

splunk-usergroups.slack.com

Dashboard studio changes:

https://docs.splunk.com/Documentation/Splunk/latest/DashStudio/WhatNew