Why Your SIEM Doesn’t Work

This is why your security information and event management (SIEM) doesn’t work:

No, it’s not Gartner’s Magic Quadrant. It has to do with a highly disproportional ratio between benign and malicious events that are collected and processed. Every event that is generated by SIEM systems in an IT environment can indicate either a benign or a malicious activity.

This creates four possibility:

1. True positives (TP): Truly malicious events that the SIEM identified as malicious.
2. False negatives (FN): Truly malicious events that the SIEM identified as benign.
3. False positives (FP): Truly benign events that the SIEM identified as malicious.
4. True negatives (TN): Truly benign events that the SIEM identified as benign.

Applied to the quadrant, these four possibilities will be distributed as such, with the Y-axis representing the true state of the event and the X-axis representing what the SIEM indicated:

Now, let’s consider the ratio between malicious and benign events in an IT environment – my conservative estimate puts this figure at 1:1000. Such a ratio would mean that a security operations centor (SOC) that collects 5,000 events per second will have to handle five real security events every second, or 300 per hour. If this is the case, every SOC would have to increase their headcount by at least a factor of 10, and all the organization would be doing is chasing down these incidents.

In the context of the quadrant, for every 100,000 events, 100 will be malicious while 99,900 will be benign. This means that the 100 malicious events will be distributed across the two sections of the malicious column (left), and the 99,900 benign events will be distributed across the two sections of the benign column (right). How these events will be distributed within the columns depends on the quality of the SIEM’s content, or correlation rules. Let’s assume you have a team of superstar SIEM engineers who created content that can correctly identify 99 percent of the malicious events and correctly ignore 99 percent of the benign events. Applying these detection capabilities to our 100,000 events will result in:

Now here comes the interesting part. The first row has all the events the SIEM indicated are malicious. Some of them are truly malicious and some are benign. The ratio between the events that are truly malicious and all the events the SIEM identified as malicious is 99/(99+999) which is about 9 percent. This means that when the SIEM identifies an event as malicious, there is only a 9 percent chance of it being truly malicious. That’s not a great track record when starting a security incident, especially when it requires asking for remediation measures and possibly affecting the organization’s productivity. Remember also that this is based on a SIEM that that can correctly identify 99 percent of malicious and benign events. If the detection rate is not as good, the chances of an incident being truly malicious drop.

How can this be? The answer is that this is the effect of the highly disproportional ratio between benign and malicious events. In such cases, very good accuracy (99 percent, in this case) is not even good enough.

What can be done? We have to turn the SIEM cybersecurity funnel on its head and use the vast majority of benign events to learn what is normal, rather than treating this data as simply stuff that should be discarded. Not using normal activity to learn what’s abnormal means your SIEM throws away knowlege every single day. Exabeam uses the normal activity to learn what’s abnormal and truly malicious.