Baselines creation

Data selection

An example of our algorithm learning a cyclical behaviour

Among all the data available we select those that are useful to describe the current situation: we give highest weight to most recent data and to those containing recursive behaviour (daily, weekly, monthly).

There are often cycles and seasonal behavior in the data, and therefore there are historical data that have a higher weight when baselining. For example in many cases the behaviour 24 hours ago is more representative of the situation now than the behaviour 13 hours ago (daily cycle). Or the behaviour one week ago is more significant than the one 5 weeks ago. The current version is using a static selection, the same for all the time series, with the same weight. This is very light weight as we don’t have to learn the weights for each metric, and makes this model easily scalable. Common sense is used for this selection: recent data matter more and we use daily, weekly and monthly cycles. This makes the model an equivalent to a hybrid between clustering and a static ARIMA.

Baseline creation

High Frequency High Activity

This are the metrics that are richer of information, so we can make a more complete analysis. Before forming the baseline the data are re-aggregated considering the frequency, to treat nulls and reduce noise.

Clustering for High Frequency High Activity

This is the part that groups the data observed in the past in different observed behaviours. The historical data are grouped in up to three different clusters. One describing the most frequent behaviour and up to two secondary behaviors, which includes less frequent behaviours or past anomalies. Therefore the approach we use is a multi-baseline behaviour, in which one baseline corresponds to the main behaviour, and the rest describe secondary behaviours which cannot be guaranteed to be anomaly free. To do this we use univariate unsupervised clustering methods. Creation of the baselines

The baselines that are going to be used in the anomaly detection are created. We use different models for the three different groups of time series classified in the pre-processing step.

One baseline to describe the most frequent behaviour is always formed, secondary baselines are created if there are enough data that do not fit the main behaviour. This is done by fitting per hour the historical data and considering 3 standard deviations.

An example of a time series (the line in blue) compared to the baselines. The main baseline is in light blue, while the secondary baselines are in shades of grey.

Trend study for High Frequency High Activity

An autoregressive model is learned to predict the next data point based on the last data point which came in. This autoregressive model is used in the anomaly detection phase to confirm that the trend of data points diverging from the main behaviour is to be considered anomalous.

Confirmation window for High Frequency High Activity

For these metrics we use a confirmation window of 15 minutes and an anomaly is detected only if there are more than 8 minutes in which the value of the metric deviated from the main baseline. This metrics are prone to frequent oscillation, so using a confirmation window reduces alert fatigue. This means that anomalies that are shorter than 8 minutes are not detectable.

High Frequency Low Activity Low Frequency

In the case in which the data are aggregated as total the analysis is considering the data smoothened by re-aggregating by 5 minutes, this is to reabsorb oscillations and distinguish the case in which one metric is consistently zero for long time from the one in which there are occasional oscillation to zero. A single baseline describing the most frequent behaviour is formed. No autoregressive model is considered. This is because this have way less structure than the HFHA one.

Low Frequency

The data are aggregated hourly, this is because this tipe of metrics have only hourly anomaly detection enabled. A single baseline describing the most frequent behaviour is formed. No autoregressive model is considered. For all the three considered cases (HFHA, HFLA, LF) the aggrgation of the metric is taken into considetation when selecting the algorithm..