Contextual Anomaly Detection

Contextual anomaly detection is the analysis of data (usually in time series format) to detect anomalies, specifically by combining the main dataset with other data points to understand their meaning in context. It is a subset of general anomaly detection.

Contextual anomaly detection often requires a mix of real-time processing of the main dataset together with batch processing of historical contextual data to properly interpret the meaning of the original data points and effectively identify anomalies. Pathway is built to handle this kind of mixed realtime/batch workload.

A Real-World Example of Contextual Anomaly Detection

You work for a shipping company. Let’s say you have a freight container with a refrigerated section which is monitored by an IoT device. This IoT device measures the temperature and sends out an alert if the temperature rises above a certain threshold value. Now, it is very possible that in the course of the container’s journey the temperature might rise in the refrigerated section for completely legitimate reasons that do not require any kind of costly intervention. The door might be opened at a customs inspection, for example, or the contents may have been off-loaded and refrigeration no longer needed. This means you can’t depend on just the IoT data coming from the temperature sensor but need to perform IoT data analytics on this data point within the context of other data points about the journey, the cargo, and the environment. This is called contextual anomaly detection and requires a mix of real-time processing of the IoT data together with batch computations performed on the available contextual data.