The What and How of Evaluating UEBA Under the Hood

User and entity behavior analytics (UEBA) is a fascinating topic for security researchers. At its core, it is about finding interesting anomalies worthy of investigation from network and security logs. While problems abound in mathematical concerns, turning UEBA into a product faces even more challenges.

This blog series is a deep dive into UEBA, from the perspectives of this data scientist who has the fortunate opportunity of seeing and making UEBA a commercial reality. I’ll show the components of UEBA and how raw logs are transformed into threat cases. The first of this multi-part blog series begins with the topic of UEBA technology evaluation.

Before we describe how to evaluate UEBA, the question to ask is why we even bother with UEBA in the first place. According to the 2022 Verizon Data Breach Investigations Report, 93% of breaches are due to compromised credentials. For such insider threats, there is no other alternative to UEBA. You can install layers of metal doors, each with triple locks, to the castle and not a single fly can get in. But they won’t do any good if the bad guy has the keys to the kingdom. While threat prevention technologies are a necessary component of enterprise security, companies are recognizing that UEBA detection tools are an integral part of security defense. The recent Lapsus$ attack is a case in point: the bad guys transacted with employees to purchase their credentials. Once inside the network, only anomalous behavior detection with UEBA has a chance of detecting the activities early to minimize the impact.

Without the familiarity with statistics and machine learning that support UEBA, how does one evaluate vendors’ technologies? There are three dimensions to consider:

Accuracy

Security vendors say little about the accuracy performance of their products, partly because it is challenging to have objective metrics. In my days of research in speech and image recognition, standard benchmark datasets were available for machine learning model research. They allow researchers to collaborate and compare results. This is how the field makes systematic gains over time. In contrast, in cybersecurity research, aside from some small academic synthetic datasets, there are no standardized large-scale, real-world datasets available for vendors or academic institutions to collaborate with. As a result, each vendor is on its own to claim the superiority of their product. But, do not despair if you are in a position to evaluate UEBA products.

Here are some aspects to consider:

• Although UEBA goes beyond signature-based detection, it is also prone to some irrelevant alerts. It is critical to minimize effects from noisy anomalies to allow true threats to surface more easily. Check with the vendor and make sure the product has the means to explicitly address false positives — for example, another layer of meta learning on the raised anomalies.

• Some anomalies are stronger risk indicators than others. Weaker anomalies do not mean they are less useful. The presence of related anomalies, even if individually they are weak, together can strongly indicate risk. Ask how anomalies are combined to provide detection signals.

• Context is king when it comes to investigation. But IT environments are highly dynamic and every environment is unique. See if the product can adaptively figure out on its own the contextual data needed for risk calibration. For example, context like whether an asset is from an executive is useful to calibrate risk of an abnormal access event. Similarly, identifying the user’s peer group is a required context before peer-based analysis. Static peer labels from Active Directory (AD) can’t capture the users’ dynamic social behaviors. Behaviors observed through data bst determine user affiliations.

In short, detecting interesting results with minimal noise is the goal of UEBA. Understand and evaluate what a risk engine does under the hood; in other words, know what you’re getting.

Usability

Ease of use is important for any resource-constrained security operations center (SOC). As UEBA surfaces cases of interest, cases must come with a high degree of explainability, both to support a quick triage experience and to reduce the later manual investigation effort. In outcome explainability, outputs from a white-box modeling approach are definitely more preferable than those from a black-box modeling approach like deep learning. Demand that your UEBA give you plenty of context on why a particular case has a particular risk score. A high-scoring case presented without tangible explanation is a hard sell for an analyst to act on. In addition, check if the tools can generate additional insights relevant to the raised cases; for example, if an anomaly alert is raised on a user account, be sure the product can provide information on what the user’s normal behavior is like with respect to the anomaly. Or ask if the product has ways of sensibly linking anomalies to provide insights.

The goal is to automate outcome explainability and insight generation. Otherwise, analysts who need to manually curate the information become database ninjas, spending less time analyzing, but more time searching and constructing, complex database queries.

Ease of use is also about operational concerns. There will be times when behavior model training needs to be restarted from historical data, either due to event parsing correction or an updated modeling method. Ask the vendor if the restart is seamless for the real-time operation, or if some system downtime is necessary, which is undesirable. Another operational consideration is on processing of late-arriving events. Late-arriving events are a fact of live in data ingestion, either due to the network delay or security devices sending only batched and delayed events. Understand how the system responds to these events, whether discarding them outright or retroactively processing them in time.

Scalability

The volume and velocity of events on enterprise networks are ever increasing, growing from tens of thousands to millions of events per second. The behavior learning and event scoring of a product must be able to cope with cloud-scale volume. Otherwise, there will be a necessary tradeoff between accuracy and performance. Data approximation and suboptimal choice of machine learning algorithms can reluctantly be made to cope with the volume. Validate with the vendor that they satisfy your data ingestion requirements.

Conclusion

Above are my broad considerations for assessing UEBA technologies. Next in this series, I’ll go more in depth toward building a UEBA risk engine.

The Ultimate Guide to Behavioral Analytics

This comprehensive guide was created to help organizations evaluating UEBA solutions better understand it and how it can be adopted to improve your overall security posture with faster, easier, and more accurate threat detection, investigation, and response (TDIR).

Read this eBook to gain clarity on confusion about the growing UEBA market, and learn about:

• What UEBA is and why it is needed
• How UEBA is different from other security tools
• The different types of UEBA solutions
• Factors to consider when evaluating UEBA solutions
• Threat-centric use cases