PART I: AIOps 2.0 - The Evolution of Correlation
The problem of event correlation has been felt deeply from the dawn of large scale distributed computing. Throughout the previous 30 years, as networks evolved, armies of technical support personnel were required to keep an eye on event screens and respond to pages on their mobile devices relating to potential issues. The number of personnel required to watch these screens and be “on call” is greatly reduced with better event correlation. In terms of an organization’s strategy for correlating events prior to visualization and management by first-line support workers, there are three levels of maturity:
- Rules-based;
- Static Correlation with Algorithmic Support;
- Dynamic AI-based Correlation.
An association rule is a well-known strategy for exploring potential patterns, correlations, and relationships in a variety of datasets and other data sources. Measurement/context and selection are two of the most difficult aspects of developing association rules. How do we determine the degree to which our goal data is related? How do we choose the groups / clusters that an algorithm or experience has identified? We normally employ data mining approaches when dealing with large datasets, and in this situation, the association rules tend to provide us with a large number of rules from which to build a good model.
As a customer, were you happy to be recommended products in e-commerce stores that you didn’t even know you may need or like? If yes, there’s a whole “rule” behind it that we’d be happy to briefly explain. This rule known as “Market Basket Analysis” is a famous example of this type of data mining technology in practice (MBA). When we talk about association rules, we usually refer to MBA. MBA was one of the first areas where association mining was used. We tend to detect correlations between components that occur together more frequently than we would expect if we randomly sampled all the options.
An example of Association Rules:
Assume there are 100 customers
10 of them bought milk, 8 bought butter and 6 bought both of them.
bought milk => bought butter
support = P(Milk & Butter) = 6/100 = 0.06
confidence = support/P(Butter) = 0.06/0.08 = 0.75
lift = confidence/P(Milk) = 0.75/0.10 = 7.5
The task of developing effective association rules in an IT setting is generally referred to as event correlation. Organizations at various stages of AIOPS maturity can reduce cost and improve effectiveness of their IT organization by investing in event correlation.
UPCOMING!
In the next part of AIOPS 2.0, we’ll dig further into each of these levels and discuss how machine learning modifies its algorithms when building a model.