Live anomaly detection

The anomaly detection works considering re-aggregating the data that come in in a certain span of time. The HFHA and for the HFLA anomaly detection is updating every minute, the LF every hour. For each aggregated data point, we evaluate if there is an anomaly ongoing. Single point anomalies are then built to a unique anomaly for anomalous points close in time: we deliver an anomaly detection that gives information on how a situation develops in time.

Severety of the anomaly

For each aggregated data point, a different weight is assigned if the point is in the main baseline (in the case of HFHA can also outside but diverging at a reasonable trend and not too far, judged by using the autoregressive model), in a secondary baseline (sup, inf) or outside all the baselines. An anomaly score is built incrementally by averaging among these weights in time for as long as the datapoints are mostly outside the main baseline. This results in a measurement (the score) that gives an “at a glance” description of the anomaly and can be averaged across nodes and systems. The score is then summarised into levels of criticality, classifying the data points into red (high probability of anomaly), orange (medium probability of anomaly), yellow (low probability of anomaly) (Severity of the anomaly). Blood red is reserved for red anomalies that are substained for a long time. A yellow anomaly has a lower likelihood of being a real anomaly than a red, could also correspond to anomalies escalating or resolving. If we focus on the anomalies that are classified as red, they include the most severe deviations from behaviour seen in the past, even if they will not include all the anomalies.

Deviation from the baselines

The color code for the anomaly gives the probability for that to be an anomaly but is does not tell us how much the data are deviationg from the baseline. For this reason we have introducsed the signed numeric field closest baseline deviation. This value is calculated based on the distance of the data point from the closest established baseline, normalized by the width of that baseline. Example Calculation: If the closest baseline ranges from 5 to 15 (a width of 10), and the received data point is 35:

Deviation Score = (Data Point - Baseline Upper Bound) / Baseline Width

Deviation Score = (35 - 15) / 10 = 2

Value = 0: The data point is within the expected baseline range (not an anomaly).

Positive Value (>0): The value exceeds the upper bounds of the baseline. A higher number indicates a more severe upward deviation.

Negative Value (>0): The value is lower than the lower bounds of the baseline. A more negative number indicates a more severe downward deviation.