Detection of silenced data sources

Goal

Detects data sources that, for any reason, have stopped producing events. As this can indicate problems in firewall configuration, infected devices, etc.

Description

The model analyses data in periods of n minutes. For every period, it receives the data sources that presented activity in that interval.

This model uses dataframe operations in order to solve the problem. It keeps, for every data source:

• Every how many intervals it usually produces events.

• The deviation of this metric

• The last interval that presented activity

To create a prediction it receives the data sources that presented activity in the current interval and checks for the ones that have not presented activity in an anomalously long period of time.

Characteristics

Name	Detection of silenced data sources
Data involved	The entire data lake
Alert Generation	Produces two different types of alerts: When a data source has been muted for longer than usual it produces an informational alert. When a data source has been muted for much longer than usual produces a High alert.
Raw outputs of the model	Data source Current interval The difference in time from the last period with activity The normal difference The deviation