Cribl Stream is recognized for its ability to efficiently structure data flows while providing capabilities to mask, transform, reduce, and route data. The platform's straightforward implementation and operation make it particularly suitable for enterprise environments, which has contributed to Cribl’s rapid adoption among Fortune 100 companies globally. Cribl Stream offers immediate functionality that many organizations have desired to incorporate for a long time.

Our experience with this solution spans several years, during which we have consistently been impressed by its organizational impact. As a partner we are working together with Cribl to enhance the products themselves and for the features outside of the core functionality we have developed an open-source solution: CriblUtilities - A comprehensive tool to streamline migrations and validate configurations. Our open source solution ensures a smooth implementation of Cribl Stream within your organization.

We started our development with the core functionality of database migration. Splunk provides DBConnect as a supplementary application for Splunk Heavy Forwarder. However, the architecture of the Heavy Forwarder does not support high availability. While it is possible to construct a high-availability solution around it, Cribl Stream natively supports database queries at defined intervals with built-in high availability. Making this a quick win for organizations that are starting their Cribl Stream implementation.

All you need is a user on your Cribl API and the three configuration files from Splunk and our CriblUtilities solution takes care of the rest.

The development process required substantial testing and coding efforts. Splunk utilizes a modified implementation of TOML (Tom's Obvious, Minimal Language). The standard Python TOML module proved to be incompatible with this implementation, and requires data reformatting to ensure compatibility. The migration process transforms three distinct Splunk files (db_connections.conf, identities.conf, and db_inputs.conf) into two Cribl outputs (Database Connection Knowledge items and Database Collectors). Followed by verification and additional configuration adjustments, such as schedule activation, the new configurations are deployed to Cribl via the Cribl API.

Currently, this represents the sole migration path incorporated into the utility. It serves as a foundation for the continued expansion of migration capabilities in future development.

Above is a picture shown with a couple of steps. In step 1 the setup of CriblUtilities is started. A few required parameters are asked from the user via the prompt, this results in a ".env" file. Alternatively, you can also set this file manually. In step 2 the empty screens are shown to prove the environment is empty. In step 3 the actual migration is done. The script gives insight into what is done. In step 4 the proof is shown that the database connections and collectors are created in Cribl Stream.

Cribl Stream provides native support for GitOps, which we consider very beneficial as it facilitates a unified approach to configuration promotion, testing, and administration. The implementation process is straightforward, requiring only a single API command. Our clients typically establish a workflow wherein designated individuals review merge requests between environments to identify potential configuration anomalies.

CriblUtilities does not aim to replace these manual reviews but rather to reduce their scope. The utility can be integrated into the pipeline to perform configuration verification prior to manual review, thereby reducing reviewer workload and implementing consistent validation procedures.

Our initial development focused on YAML file linting. As Cribl utilizes standard YAML formatting, we leverage the Python YAML module to identify unsupported modifications, such as improperly formatted local changes. The prevention of improperly formatted YAML files is critical, as they may compromise Cribl Stream in production.

Establishing naming conventions for Cribl Stream configuration elements is considered best practice. We have implemented naming convention verification as an additional feature in CriblUtilities. Organizations typically maintain conventions for worker groups, sources, destinations, data routes, pipelines, and packs. As configuration types store naming information in various locations, our utility simplifies verification through commands such as:

cribl-utilities check naming --conf /opt/cribl --field sources --regex "src_.*"

For pipeline configurations, similar verification can be performed with:

cribl-utilities check naming --conf /opt/cribl --field pipelines --regex "ppl_.*"

In this example there are four worker groups created. Only one applies to the new naming convention. Which states that the name of a worker group should start with "wgp_".

In this example you see the default worker group. We created an exception for this worker group in the naming convention since this group is always there. This gives two workers groups that do not apply to the naming convention:

Use the debug tag to check whether this checks all configurations you expect. Especially filtered_fields are interesting in this case:

We have decided to launch this initiative as an open-source project due to our commitment to transparency. CriblUtilities does not transmit data to external sources with the exception of your designated Cribl environment. Our open approach enables verification of these assurances. We encourage developer participation to enhance functionality, and welcome issue reports from all users to facilitate ongoing improvement.

CribUtilitites is a comprehensive tool to streamline migrations and validate configurations.

Streamline migrations and validate configurations with CriblUtilities

In early 2024, the 1 Billion Row Challenge (1BRC) went viral. The goal was simple yet daunting: process a file with 1 billion rows of temperature measurements and calculate the minimum, maximum, and mean temperatures for each weather station in the dataset. This benchmark pushed developers to explore extreme optimization techniques. The winning solution completed this challenge in just a mere 1.535 seconds, decisively debunking the long-standing myth that Java is inherently slow.

The question I aim to answer in this blog is: how was this incredible feat achieved, and what techniques can we apply to make Java code faster?

The first and simplest approach is to leverage the ability of modern processors to handle multiple tasks simultaneously—also known as parallelization. Parallelization makes work faster by dividing a task into smaller parts that can be completed simultaneously. Instead of processing everything one step at a time (sequentially), parallelization assigns different parts of the work to multiple processors or cores.

Java offers powerful libraries for parallelization, making it easy to divide tasks and process them simultaneously. Key tools include the Fork/Join Framework for divide-and-conquer problems, Parallel Streams for effortless parallel data operations, and the Executor Framework for managing threads. Features like CompletableFuture enable asynchronous parallel tasks, while third-party libraries like Akka and RxJava support advanced concurrency. These tools simplify parallelization, boosting performance by leveraging modern multi-core processors.


The fastest solutions took this approach to the next level. A common bottleneck in parallel processing is that files are often read sequentially, even when the data is processed in parallel. Without delving too deeply into the details, the fastest developers overcame this by writing custom code to split the text into chunks optimized for the number of cores and CPU cache sizes, allowing the file to be read and processed simultaneously across all cores—fully maximizing their utilization. Understanding your architecture is key.

Working hard isn’t enough—you need to work smart. While reading each byte of every line, the data must also be understood. Since they were dealing with a CSV file, this involves identifying the separators and efficiently parsing each line into meaningful pieces for further processing.

A simple way to handle this is by using regular expressions, often the go-to approach in everyday programming. However, for extreme performance, regular expressions are too slow due to their sequential nature. To optimize processing, it’s necessary to break down each line into bytes and process them in parallel, leveraging the capabilities of modern CPUs.

The first approach is using Java’s Vector API, which enables SIMD (Single Instruction, Multiple Data) operations. SIMD allows a single instruction to process multiple bytes simultaneously, significantly boosting performance. For example, instead of checking each character one by one, the Vector API processes entire chunks of data (e.g., 128 bits) in parallel, identifying target values like commas in a single operation. This method is ideal for maximizing CPU core utilization;  however, it does have some limitations, such as incompatibility with GraalVM.

The second approach involves masks, which efficiently process data at the byte level. A mask is an array that marks positions of target values (e.g., 1 for a match, 0 otherwise) using branchless comparisons. Masks avoid the performance penalties of conditional logic, ensuring predictable, high-speed processing. While less abstract than the Vector API, masks provide a scalable and hardware-agnostic solution for parallel data parsing.

3. Efficient Data Structures – Be Organized

When processing data, temporary results need to be stored efficiently while handling the next piece of data. A commonly used data structure for this purpose is the standard Java hash map (HashMap), a reliable solution for key-value storage. However, while HashMap is versatile, it can become a performance bottleneck due to hash collisions and the overhead involved in resolving them.

To address this, some developers explored alternative storage methods. One particularly intriguing approach was creating a custom hash map with forward probing. A standard Java HashMap works by transforming a key into a hash value and storing the value at an index determined by the hash. However, hash collisions occur when multiple keys are assigned the same index. In such cases, the HashMap stores these values in a list. Retrieving a value then requires accessing the list and traversing it to find the desired value, which can slow down performance.

In contrast, a hash map using forward probing avoids storing values with the same index in a list. Instead, it searches for the next available slot in its array and places the value there. This approach is faster, provided collisions are kept to a minimum. Since the set of weather stations in this case was fixed, developers could design a custom hash function optimized to minimize collisions. By fully understanding the constraints of their data, they were able to implement extremely efficient organized data.

This approach is less about optimizing the code itself and more about optimizing how Java runs the code. As you may know, Java source code is first compiled into an intermediate format called bytecode, which is platform-independent. At runtime, the JVM (Java Virtual Machine) uses a Just-In-Time (JIT) compiler to convert this bytecode into native machine code specific to the underlying hardware. This process allows Java programs to achieve performance close to that of native applications.

However, JIT compilation happens during program execution, which can create overhead. The JVM spends time analyzing and optimizing code paths at runtime, leading to initial "warm-up" delays. This can slow down execution, especially for performance-critical or short-lived applications.

To address this, contestants used GraalVM, which improves runtime performance by leveraging an Ahead-Of-Time (AOT) compiler. GraalVM compiles Java bytecode into native machine code before the program runs, eliminating the need for JIT compilation during execution. This reduces warm-up times and improves startup performance. Additionally, GraalVM applies advanced optimization techniques to generate highly efficient native code, resulting in faster execution overall, particularly for applications that demand high performance or quick startup times.

For some developers, even this wasn’t fast enough. As you may know, Java uses a Garbage Collector (GC) to clean up objects that are no longer referenced. Without this process, a program running long enough would eventually run out of memory. However, while the Garbage Collector is working, the CPU is occupied and not running your program, which can cause slowdowns.

To remedy this, some developers simply turned off the Garbage Collector. (Technically, they used the Epsilon GC that just doesn’t do anything.) This is feasible for short-lived programs because all memory is automatically released once the program finishes execution. By disabling the GC, developers eliminated this overhead, allowing the CPU to focus entirely on the program and achieve faster execution times.

One of the key benefits of using Java over languages like C or C++ is its strong emphasis on safety. Java is designed to minimize the risk of causing harm to your system while running code. Each Java application operates within its own dedicated memory space and cannot access or interfere with memory allocated to other processes. This isolation, often referred to as a sandbox, ensures that Java programs cannot disrupt other processes or compromise system stability.

Java is also memory-safe, meaning it prevents unauthorized access to memory. Unlike C or C++, where developers manage memory manually, Java uses built-in mechanisms to handle memory allocation and cleanup. For example, Java’s automatic Garbage Collector ensures that unused objects are safely removed without risking memory leaks or invalid memory references. Combined with its robust runtime checks and strict type system, Java significantly reduces vulnerabilities, making it a highly secure choice for developers.

However, this level of safety comes at a cost: it can slow down your code. To achieve even greater performance, some developers bypassed these safety features by using Java's Unsafe library for direct memory manipulation. The Unsafe library allows developers to perform low-level operations, such as directly allocating and accessing memory, circumventing Java’s built-in memory management and safety checks.

One specific optimization involved bypassing the unmapping of data, which is typically handled automatically by Java’s memory management system. By taking control of these processes, developers were able to reduce overhead and execute operations significantly faster. 

The above-mentioned techniques sacrifice safety for speed and should, therefore, be used only in extreme cases when a developer truly knows what they are doing.

Parallelization – Teamwork: Break tasks into smaller pieces and execute them in parallel to maximize CPU utilization, ensuring no processing power goes to waste.

Efficient Parsing – Work Smart: In programming, solving problems efficiently matters. While there are many ways to achieve a solution, some paths are significantly faster—choose the shortest road to your goal.

Efficient Data Structures – Be Organized: Every data structure has strengths and weaknesses. Understand your data and select the structure that best fits the problem for optimal performance.

GraalVM – Skip Unnecessary Steps: Fast code isn’t everything; how your code is executed matters too. Tools like GraalVM can eliminate unnecessary runtime overhead, allowing your code to run more efficiently.

Unsafe – Don’t Try This at Home: In extreme cases, bypassing Java’s safety features with tools like the Unsafe library can deliver higher performance. However, this approach should only be used with careful consideration and a strong justification.

https://www.morling.dev/blog/one-billion-row-challenge/

Inside the Fastest Solutions of the 1 Billion Row Challenge

Bekijk hieronder de aftermovie van de CINQ Conferentie. De jaarlijkse afsluiting van de CINQ Sprekersopleiding was weer een feestje!

We hebben genoten van drie mooie presentaties:
🔸 'Deployen op vrijdagmiddag zonder stress' door Bas Stoker
🔸 'Wirespec vs OpenAPI' door Jerre van Veluw
🔸 'Immunologie door de lens van een ontwikkelaar' door Alex Witkamp

Aftermovie CINQ Conferentie 2025

De volgende CINQ sprekersopleiding gaat waarschijnlijk in het najaar van 2025 van start. 

Volg ons op Linkedin om op de hoogte te blijven: https://www.linkedin.com/company/cinqict/ 

Bekijk de aftermovie van de CINQ Conferentie 2025

Were you ever curious about how companies like Uber, Just Eat, and Strava show you the location of your delivery, ride, or run in real-time? Well, in this tutorial we will go over the two main ways that map data can be displayed and interacted with on a web page. 

a discussion of the two main ways to display map data on a web page, and their pros and cons

inserting actual map data into the database

spinning up a geoserver to serve the map data

spinning up a geographical REST API to serve the map data

displaying the map data on a web page using streamlit and folium


In the following video demo our resident Python Developer, James Twose, will show you how it is done!

Displaying map data on a web page: geoserver vs geoapi

We hope you enjoyed this tutorial and now have a better understanding of how to display map data on a web page.

If you have any questions or comments, please feel free to reach out to us on our social media channels or via email.

You can find the code for this tutorial here: https://github.com/jameshtwose/geoserver-vs-pygeoapi

Were you ever curious about how companies like Uber, Just Eat, and Strava show you the location of your ride, delivery, or run in real-time? In this tutorial we will go over the two main ways that map data can be displayed and interacted with on a web page.

Displaying map data on a web page: Geoserver vs geoapi

Geoserver vs geoapi

Heb je altijd al willen spreken op een conferentie maar weet je niet waar je moet beginnen? Of wil je je simpelweg je presentatievaardigheden verbeteren? Meld je dan aan voor de CINQ Sprekersopleiding! 

Na het succes van twee interne edities met onze CINQ collega’s, gaat in oktober 2024 de CINQ Sprekersopleiding open voor het publiek. 

De opleiding bestaat uit vier avonden waarin je wordt klaargestoomd voor je eerste optreden. In oktober, november, januari en februari zijn de opleidingsavonden en in maart vindt de CINQ Conferentie plaats: daar ga jij je presentatie voor publiek geven! 

hoe schrijf je een goed voorstel (paper)?

hoe maximaliseer je je kansen om geselecteerd te worden?

Tijdens de avonden word je voorzien van de benodigde kennis, en ga je vooral ook veel oefenen met presenteren. De deelnemers gaan elkaar beoordelen en van feedback voorzien. Aan het einde van het traject heeft iedereen een goed beeld van wat er nodig is om op een conferentie te kunnen spreken. Niet alleen het spreken zelf, ook alle stappen die voorafgaan aan het spreken op een conferentie heb je dan onder de knie.

We sluiten het traject af met de CINQ conferentie waarbij je je nieuwe vaardigheden in een presentatie voor publiek gaat laten zien. Hierna blijft Bouke beschikbaar als je coach en zal hij je begeleiden naar je eerste “optreden” als spreker op een conferentie.

Deelname aan de CINQ Sprekersopleiding kost €250 exclusief btw. Hiervoor krijg je:

vier avonden opleiding inclusief avondeten

training en begeleiding door een ervaren internationaal spreker: Bouke Nijhuis

begeleiding na het traject naar je eerste conferentie

Klik hier om je aan te melden. Je zit nog nergens aan vast. Na je aanmelding nemen we zo snel mogelijk contact met je op. 

Bekijk een impressie van de vorige CINQ Conferentie:

CINQ CONFERENTIE AFTERMOVIE 2024

Heb je altijd al willen spreken op een IT conferentie maar weet je niet waar je moet beginnen? Of wil je je simpelweg je presentatievaardigheden verbeteren? Meld je dan aan voor de CINQ Sprekersopleiding!

Meld je aan voor de CINQ Sprekersopleiding

CINQ Sprekersopleiding

Kun je al (een beetje) programmeren en ben je benieuwd naar Python? Kom langs en leer van Bart van Dijk waarom Python een fijne taal is om snel mee te kunnen developen.

De workshop
In de workshop leer je over de datatypes binnen Python en een aantal geavanceerdere concepten. Vervolgens laat Bart zien hoe je Clean Code kunt schrijven in Python.

Als afsluiter kan er gekozen worden tussen een set opdrachten voor Object-Georienteerd Programmeren, Functioneel Programmeren, of algoritmische opdrachten a la Advent of Code.

Voor wie is deze workshop?
Developers, DevOps- en Data-engineers met al enige ervaring in programmeren.

Hoe kan ik me voorbereiden?
Zorg dat je je favoriete IDE (bijvoorbeeld VSCode) paraat hebt die Python ondersteunt. (Of Notepad als je van hardcore code schrijven houdt)

Het programma
17.00u - inloop met eten
18.00u - start workshop
20.00u - borrel / netwerken

Het is een workshop, dus neem je laptop mee!

Kom naar de Python workshop op 10 september

Splunk 9.3 has been released and  there are some new features that we would like to share with you. In the release notes you can find all the features added to this release.

Dashboard Studio - Scheduled PDF and PNG export

It is now possible to schedule PDF or PNG exports for Dashboard Studio dashboards.  For users to be able to do this you have to give them the ‘list_settings’ capability. There are a few limitations that you should consider, see the Splunk Documentation for the details.

A nice bonus is the ability to use the API to fetch a PNG, this means that you can use your automation tooling to fetch a PNG of a dashboard and display that on a website. This is much easier than using embedded reports.

Instead of Python 3.7 the new default is now Python 3.9, you could override this in your app settings but we highly recommend testing all your Python scripts in a test environment with Splunk 9.3 before upgrading on production. 

On busy search heads it is now possible to apply a memory reduction measure for the Splunkd process. In the Splunk conf description it does state that you should consult Splunk support before enabling this setting. 

If you used role-based field filters then you should know that after upgrading these filters will no longer work. You might want to check out the new field filter option, but keep in mind that you can no longer use the following commands: mpreview, mstats, tstats, typeahead, and walklex

If you use Splunk Enterprise Security you probably won’t be able to use this since Splunk Enterprise Security relies heavily on tstats. For more information click this link. 

Indexer cluster rolling upgrade automation

If you have not yet automated your Splunk indexer cluster upgrade you might want to check out this app that helps you. Do note that using systemd for Splunk is currently not supported.
https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/AutomatedIDXCrollingupgrade

Splunk 9.3 has been released and we highlight some of the new features in this blogpost.

Splunk 9.3 new features explained

Splunk 9.3 - new features explained

Ieder jaar trekken we er met onze collega's op uit voor een actief uitje. Dit keer kwamen we samen op het mooie terrein van de Middengaarde in Nieuwegein. 

Hier werden we getrakteerd op een heerlijke barbecue, leuke activiteiten en als bonus mochten we ook op het terrein blijven overnachten. Na een gezellige avond / nacht bij het kampvuur konden we zo naar onze tent kruipen. De CINQ Camping, voor herhaling vatbaar? 

CINQ EXPERIENCE 2024 - Aftermovie

Wil jij er de volgende keer ook bij zijn? Check dan onze vacatures

Aftermovie CINQ Experience

Ever wondered how your API would perform under heavy load? Load testing is a way to simulate a high number of users accessing your API at the same time. This can help you identify bottlenecks in your system and help you optimize your API for better performance.

Well luckily enough one of our data engineers, James Twose, will give a video demo on how to use the open source Python framework Locust to load test your API.

Locally run a postgres database and a fastapi-sqlalchemy API using docker-compose

Loadtesting your API with Locust

We hope you enjoyed the demo and learned something new about load testing. If you have any questions or feedback, feel free to reach out to us on our social media channels. You can find the code for this demo here: https://github.com/jameshtwose/api-framework-comparison

In this video data engineer James Twose shows you how to loadtest your API using Locust.

Wat ooit is begonnen als een hackathon bij DHL, één van onze klanten bij CINQ ICT, is nu een framework aan het worden om Azure resources aan te maken en te beheren. Het idee was om de Infrastructure as Code weg te abstraheren bij de ontwikkel teams, maar om de resources via configuratie files (bijv. YAML) aan te kunnen maken. Deze manier van werken zou ook een eerste stap kunnen zijn naar een vorm van self-service.

Terraform is een van de meest gebruikte tools om infrastructuur in te richten vanuit code (IaC). Het gebruikt hiervoor een taal genaamd HCL (Hashicorp Configuration Language). In deze blogpost onderzoeken we hoe we het aanmaken van resources kunnen configureren in YAML files en wat de voordelen hiervan zijn.

Een van de beste eigenschappen van YAML is de afwezigheid van “syntax overhead”. Hiermee kunt je key value pairs overzichtelijk opschrijven. Laten we eens kijken naar een vergelijking van bepaalde HCL-code en YAML.

locals {
  storage_accounts = [
    {
      name                     = "storageaccountname"
      account_tier             = "Standard"
      account_replication_type = "ZRS"
      access_tier              = "Hot"
      containers               = [
        {
          name                  = "vhds"
          container_access_type = "auto"
        }
      ]
    }
  ]
}

storage_accounts:  - name: storageaccountname
    account_tier: Standard
    account_replication_type: ZRS
    access_tier: Hot
    containers:
      - name: vhds
        container_access_type: auto

Zoals je kunt zien, is het verschil in aantal regels behoorlijk groot. Natuurlijk zal dit veranderen zodra we wat HCL-code toevoegen om de YAML-configuratie te importeren, maar dit wordt snel groter als de infrastructuur groeit.

Het laden en converteren van een YAML-bestand naar HCL is heel eenvoudig. Je kunt het op één regel doen met behulp van de yamldecode en file functies:

locals {
  config = yamldecode(file("config.yaml"))
}

Na het inlezen van alle yaml files is er een Terraform map beschikbaar met alle resource-types en de in de yaml geconfigureerde properties.

config = {
  "keyvaults" = [
    {...},
    {...}
  ]
  "storage_accounts" = [
    {...},
    {...}
  ]
}

De "router" module is de orchestratie module en de spin in het web die meerdere resources en/of resources-typen aan kan maken. De router module leest alle yaml files in en roept voor iedere resource-type de betreffende resource module aan. Als er in de yaml file meerdere resources van hetzelfde type worden aangemaakt dan zal middels een for_each loop de resource module meerdere malen worden aangeroepen.

module "keyvaults" {
  for_each = {
    for resource in lookup(config, "keyvaults", []) :
    resource.name => resource
  }
  source = "../../microsoft.keyvault/vaults"
  keyvault_info = each.value
}

Om de resource module folder structuur te geven, gebruiken we het resource-type als folder naam.

modules/
├── internal
│   ├── global
│   ├── router
├── datastax
│   └── astradb
├── microsoft.cache
│   └── redis
├── microsoft.datafactory
│   └── factories
├── microsoft.dbforpostgresql
│   └── flexibleservers
│       ├── databases
├── microsoft.eventhub
│   └── namespaces
│       └── eventhub
├── microsoft.keyvault
│   └── vaults
├── microsoft.sql
│   └── servers
│       └── databases
├── microsoft.storage
│   └── storageaccounts
├── microsoft.synapse
│   ├── spark_pool
│   └── workspaces
└── microsoft.virtualmachine

De resource module zal de daadwerkelijk resource aanmaken en bevat alle validatie regels op de input parameters die noodzakelijk zijn. Verder zal de module verplichte parameters afdwingen en "waarden van gezond verstand" gebruiken voor niet verplichte niet opgegeven properties.

variable "storageaccount_info" {
  type = object({
    name                          = string
    account_tier                  = string
    account_replication_type      = string
    access_tier                   = string
    is_hns_enabled                = optional(bool, false)

    containers = optional(list(object({
      name                  = string
      container_access_type = optional(string, "private")
    })), [])

    tags        = optional(map(any), {})

    validation {
      condition     = contains(["Standard", "Premium"], var.sa_info.account_tier)
      error_message = "Invalid account tier. Valid options are 'Standard' and 'Premium'."
    }

    validation {
      condition     = contains(["LRS", "GRS", "RAGRS", "ZRS", "GZRS", "RAGZRS"], var.sa_info.account_replication_type)
      error_message = "Invalid account replication type. Valid options are 'LRS', 'GRS', 'RAGRS', 'ZRS', 'GZRS' and 'RAGZRS'."
    }

    validation {
      condition     = contains(["Hot", "Cool"], var.sa_info.access_tier)
      error_message = "Invalid access tier. Valid options are 'Hot' and 'Cool'."
    }
}

Ook zal de resource module best practices implementeren (denk bijv. aan privacy-by-design principes zoals private endpoints, encryptie, logging) zonder dat dat in de yaml files is aangegeven. De maker van de yaml file hoeft alleen maar de meest voor de hand liggende properties in te vullen zoals bijv. de naam van de resource en de uitvoering (SKU, pricing tier).

YAML-bestanden zijn eenvoudiger dan Terraform locals en/of tfvars-bestanden voor het beheren van niet-triviale Terraform-configuraties. Omdat development teams nu zelf YAML files kunnen aanmaken is dit een eerste voorzichtige stap naar een vorm van self-service.

Binnen de DevOps-unit van CINQ ICT gaan we dit jaar een Terraform hackathon organiseren om te kijken of we dit concept nog verder kunnen uitwerken. Zo heeft iedere resource module een scope van alleen zichzelf en heeft dus geen weet van het bestaan van meerdere en/of andere resource-typen.

Een belangrijk onderwerp van de hackathon zal dan ook zijn hoe we een integratie laag tussen de verschillende modulen kunnen realiseren. Denk daarbij bijv. aan hoe koppelen we een keyvault aan een Kubernetes namespace, of een storage account aan een Synapse Workspace, hoe gaan we om met firewall rules? Zonder daarbij de code in de modules heel complex te maken.

In dit blog lees je hoe we met behulp van yaml files Terraform resources kunnen configureren.

Metadata driven Terraform deployments

In this blog I’ll demonstrate how to build a Splunk XML Dashboard by using AI only. Following a case study, I will guide you step-by-step from the initial concept to a fully developed dashboard. 

For this exercise, I used the Cinq ICT website as a base and reference for the layout and aim to create a dashboard that can serve as the homepage for a Cinq ICT Splunk app. For the dashboard code generation, I’m making use of so-called Large Language Model (LLM) bots. In this case chat GPT, the 3.5 free version. I’ll also use the Dall-E 3 image generation via Microsoft CoPilot to generate some logo images for our Data, DevOps, and Development Unit. Feel free to use the prompts in this case study in combination with other LLM bots to see the differences in the image or code generation.

Below I visualized the concept for the Splunk dashboard. The goal is that AI creates the code for each dashboard component, based on 4 prompts.  

1. At the top, we have a header that includes an image with accompanying text below and on the right. 
2. Below that, there will be a headline row “App Dashboards”, a copy of this code will be used to create the “Links/Help” headline.
3. Between these 2 headlines we have 4 tiles with the dashboard 1/2/3/4 links
4. At the bottom, a list of URLs. 

The colors indicated in the color schema are picked from the cinq website and will consistently be used in the AI prompts. The blue circles in the header will be replaced with the generated images.

When asking things to a LLM bot like ChatGPT it’s important to take the following tips/tricks into account for better responses.

1. Be Precise: Clear, detailed inputs yield better responses.
2. Simplify: Break complex questions into shorter ones.
3. Follow-Up: Ask clarifying questions if needed.
4. Provide Context: Background information enhances relevance.
5. Highlight Keywords: Emphasize important terms to focus the response.
6. Avoid Ambiguity: Frame questions clearly to reduce misunderstandings.
7. Confirm Understanding: Summarize responses to check accuracy.

The general idea is to leave few to no room for interpretation, as that will cause inaccurate responses. 

As shown in the concept above we need 4 prompts to generate the dashboard. We’ll take it top-down and begin with the header, applying the tips outlined above. I’ll write the below query in chatGPT. Based on my experience with AI and Splunk code dashboard generation, I know that I need to explain a few more details to get the right response. I already instructed chatGPT to change the <dashboard> tag by including version 1.1, a light theme, and a hidden title. I also ensure to use self-closing <br> and <img> tags. Finally, I eliminated the grey spaces between rows using CSS. If you don’t know – and therefore may not apply - these instructions you will get code that does not work properly in Splunk and then you need to manually apply these fixes in Splunk, also a good exercise. 

Prompt 1 - Header
Please create a header for a Splunk XML dashboard using HTML/CSS. I need a white background with rounded corners with 3 mid-left-aligned image placeholders from left to right (logo_1.png, logo_2.png, logo_3.png) in the Splunk Search app directory, 350px wide and 100px tall). 8px padding-bottom, Below the image, include the text "Data | DevOps | Development" in #575756, with the '|' in orange (#F39201), all bold and 24px. On the right, place the text "CINQ ICT" in bold, 48px, #575756, padding-right 80px padding-botton 12px, with a smaller subtitle "Net iets meer | net iets anders" in #ABABAB, 26px, no padding. Use <dashboard version="1.1" theme="light" hideTitle="true"> to start, and ensure all image and br tags are self-closing. With CSS set the .dashboard-header class min-height to 0px; and the .panel-body.splunk-html padding to 0px;

Optional Prompt: Images
Generate a logo, a circle shape that symbolizes Data Analytics using only colors #58ADDF , #575756, #F39201.

For the 3 images I have used the above prompt, changing the text “Data Analytics” to “DevOps” and “IT Coding/Development”. Using Microsoft Designer, I Downloaded the images and placed them locally in Splunk search app folder src="/static/app/search/data.png".

The first prompt with the images gives the following result, note that it will slightly differ each time based on what chatGPT generates. It looks as follows:

Prompt 2 - Headlines
For an existing Splunk XML dashboard I need a new row that creates a small headline with rounded corners. A white background and a text “App Dashboards” in blue #58ADDF, bold and 16px, aligned left. So start and end with the row tag, omit the dashboard tag.

Note that I instructed chatGPT to clearly only add the row tag here, so we append to our previous code. I copy the rows 2 times in Splunk, so I have both headline rows. The code is now as follows and the dashboard looks like this:

Prompt 3 – Tiles
For an existing Splunk XML dashboard I need a new row with 4 clickable tiles from left to right with height 75. Each tile must have rounded corners, a white background and the text “Dashboard” with a number 1,2,3,4, text in blue #58ADDF , bold and 20px, aligned centre. The tiles need a hover and onclick to another Splunk dashboard.  The first tile needs no margin on the left side, and the last one needs no margin on the right side. All style attributes are in separate style tags. Start with the row tag and omit the dashboard tag.

I add the row prompt result from chatGPT between the 2 headline rows, to add the 4 dashboard link tiles. The result looks as below. Note that the hover function is also added and even came with some advanced CSS transitioning for a smooth background change. 

Prompt 4 – Links/Help
For an existing Splunk XML dashboard I need a row with a with background and rounded corners. In this row I need a list of URLs underneath each other with mouse hover functionality. The link URLs are: “Home”, “What we do – Data”, “About CINQ”, “Events”, “Podcast”, “Blog”, “Vacancies”, “LinkedIn - Tom de Bruijn”. The URL text in blue #58ADDF , bold and 16px, aligned left. Start with the row tag omit dashboard tag.

After Prompt 4 The final code results in this and looks like below. 

I hope you found this case study informative and enjoyed reading the blog! If you'd like to connect, feel free to add me on LinkedIn at https://www.linkedin.com/in/tcdebruijn/. I'm always more than happy to chat or help out if needed. 

In this blog Tom de Bruijn demonstrates how to build a Splunk XML Dashboard by using AI only. Following a case study, he will guide you step-by-step from the initial concept to a fully developed dashboard.

Build a Splunk Dashboard with ChatGPT or AI only

Build a Splunk Dashboard with ChatGPT

Vakmensen, dat zijn we. En wat we doen op het gebied van informatie- en communicatietechnologie helpt organisaties slim verder. Effectief en meetbaar bovendien. Aangenaam, CINQ. Specialisten in Data, DevOps en Development. Wij geloven in specialisten, vandaar de driedeling Data, DevOps en Development binnen ons bedrijf. Zo zorgen we dat jij excelleert, want je werkt het best waar je je het meest thuis voelt. Logisch, toch?

cinq_logo.png

cinq_logo_square.png

CINQ ICT

We delen graag onze kennis over Data, DevOps en Development met je. Op ons blog lees je tips en tricks over bijvoorbeeld Splunk, Docker, Kubernetes en Java.

Onze laatste blogs