Realtime anomaly detection with Trino

Have you ever been curious to know how to use streaming data with anomaly detection algorithms to check if the system you care about is doing anything peculiar?

This could be detecting fraud in your banking software, disease detection in healthcare, maintenance of industrial hardware… Well in this instance one of our Data Engineers, James Twose, will take you through an anomaly detection project using mock voltage data taken from a fictional smart meter.

He will explain:

• What technology can be used to do this, (Kafka, Trino, PostgreSQL, River, sklearn, Streamlit)
• How all of the different parts interact with each other (mainly focussing on the use of Trino as a semantic layer between everything)
• What the idea of anomaly detection is, and will lightly touch on how it works and,
• How you could run this locally on your own computer, with an in depth demo of the system

We hope that what you learned from this was useful, and that it shone a light on the possibilities of real time anomaly detection. We are curious what use cases you come up with, and fingers crossed, that this is a good starting point for your projects.

Find the full repository on GitHub: https://github.com/jameshtwose/power-anomaly-detection/blob/main/README.md