Security Data Lakes: Comparing the “Do It Yourself” Deployments Versus Commercial Solutions

Hackers are experts at concealment and manipulation, and they’re constantly evolving—finding new ways to achieve their aims. When a breach has occurred, often organizations aren’t even aware of it. Frequently, organizations don’t have the data to understand what has happened or they may not have the right resources and talent to respond. Or, perhaps, they can’t perform the right search and analysis of their data.

Cybersecurity operations doing log management often aren’t keeping pace with the rapid changes in data growth, new data types, cloud architectures, and open-source, big data management. And now, log volume is measured in petabytes—with data streaming in from internal networks, endpoints, and the cloud—requiring powerful and advanced systems to keep up.

Legacy data management and analytics solutions simply haven’t been designed for these conditions. As data grows, security systems are licensed by the byte and the cost for solutions is increasing every year. As a result, many CISOs are spending more for their security budget but without significant gains.

There are a number of vendors who are providing log management and various security analytic technologies. There are also open source “Do it Yourself” (DIY) log management technologies such as Elastic Stack (also referred to as ELK) that organizations can leverage, deploy, and use to manage their environment. But can the DIY approach effectively address your cybersecurity challenges and requirements?

In this blog we look at the challenges of using a DIY deployment using open-source technologies like the Elastic Stack versus deploying a commercial security data lake that leverages ELK. But first, here is some background on using a specialized log management solution for your cybersecurity solution.

Why a specialized security data lake is the optimal solution for cybersecurity use cases

A security data lake is a specialized data lake designed to fulfill cybersecurity use cases, and ingests, analyzes, and visualizes log data for analysts. Security data lakes are designed for log data growth and the complexity of cybersecurity analysis. Security products (firewalls, VPN, DLP, proxies, etc.), network devices, endpoints, and servers all create their own logs. Without a data lake, security analysts must access these logs from numerous sources to perform their investigations.

What is the Elastic Stack?

The “Elastic Stack” is a combination of Elasticsearch, Logstash, and Kibana, which are open source tools that are the foundation of a data lake log management system.

Elasticsearch is a distributed data store where data can be searched quickly, allowing for advanced queries that give developers opportunities to perform detailed analysis.

Logstash is the data ingestion tool that sits in front of Elasticsearch which collects data from various sources. Logstash supports a wide range of data types and ingests data from a multitude of sources—like systems, web servers, firewalls, access, databases, and applications—and it can collect the data all at once.

Kibana is a visualization layer that works on top of Elasticsearch.

The challenges of deploying a “Do it Yourself” ELK

While developing a data lake using ELK, our development team ran across a number of issues that any DIY team will face. Let’s take a closer look at the issues:

Needs fine-tuning
First and foremost, while ELK is built for scalability, it also requires fine-tuning and optimization to ensure high performance. Defining the memory, sharding, and search settings is a long trial and error process.

Lacks data enrichment
Data enrichment with context is absent in ELK. Data enrichment involves the engineering effort to write connectors to Active Directory/LDAP, or a Human Resources Management (HRM) system and applying context to the data. Context is important for effective threat detection, and analysts need context to perform their investigation. Details like a user’s location, device type, and job role help analysts to understand what the user may be attempting and whether there might be a legitimate reason for accessing certain systems or data. For example, a sales team employee who accesses a server in finance might cause alarm, unless that person is in sales operations and calculates commissions.

Upgrade challenges
Upgrading a medium to large ELK deployment is a daunting task that must be managed properly and requires attention to the data replication—this becomes complex with a multi-node cluster deployment.

Operational challenges
ELK does not have built-in support for hot/warm/archive/cold nodes, which need to be designed and engineered. Hot, warm, and archive nodes within the cluster is a way to separate ingestion and retention of data. Hot nodes store the most current data for faster search. Warm/archive nodes are for data retention, which can be used during incident investigations. In addition, a production-ready solution requires a complete monitoring layer for health checks. DIY ELK often fails when not configured properly, and will fall short in predicting failures.

ELK is not free
A production-ready ELK deployment is not free. First you must consider infrastructure and storage—ELK can be deployed on AWS or on-prem. Depending on the amount of data you ingest and retain, the costs can add up. Consider the number of nodes in a cluster, the amount of data you are ingesting per day, the number of concurrent users and searches you must support, and whether you require high availability and disaster recovery clusters—all of which will increase infrastructure costs. Next, consider the Elastic Stack’s ongoing operations or support costs, which will require full time resources to set up, upgrade, and maintain. Gartner echoes many of these concerns:

“There is some feedback from customers that the workload involved in engineering these solutions [DIY ELK] to scale and the development effort to support the required event sources and analysis is significant, despite the software itself being free. This may negate the objective of being less expensive than a commercial SIEM deployment [Security Data Lake].”
– Gartner SIEM Report 2017

There are a number of challenges in deploying a DIY Elastic Stack. Carefully evaluate whether your team will be realistically able to deploy a DIY ELK project. If not, consider a commercial solution with dedicated support, professional services, and customized solutions for your use cases.

Advantages of a commercial security data lake

• Highlight Security Data – The security data lake can leverage ELK features and customize the stack to provide security use cases per the business requirements. The commercial product adds new parsers – logs, network, endpoint, etc. – into a security information model and formats records based on their types, highlighting the most relevant fields for security teams. This makes it easy for security analysts to quickly find the information they’re after, instead of spending their time visually parsing through lines of dense event logs.
• Extensible Architecture – Organizations can start small and expand as they grow. ELK is built to scale horizontally out of the box. A commercial technology can be made more resilient by allowing organizations to set up multiple nodes and query each of them independently or in combination. The security data lake can be bolstered with hot-warm architecture for faster search on recent data (hot node) and archived data (warm node). A monitoring layer for health checks can be added to make it easy for administrators to watch for any failures in a multi-node architecture.
• Effective Investigation Tool – SOC teams are able to collect, parse, and view data to troubleshoot and resolve complex security problems. Security data lakes use collectors to fetch data from security products, cloud apps, and others legacy SIEM vendors. Security related content is provided such as pre-built reports, dashboards, and visualizations to handle the common security and compliance requirements. The data is also enriched with context, which is necessary for effective threat detection. Contextual data helps SOCs to search for data more effectively that allows for better investigations of security incidents.
• Integrations to Help SOC – A security data lake is built with natural rule builder for correlation rules, dashboards, parsers to collect logs from multiple data sources and has a distributed search capability to aid security investigations. It can also have tight integration with the other solutions like behavior analytics and incident responder products, which will help your organization’s threat exposure, lower staffing requirements, and streamline your SOC’s operations.

Figure 1: The architecture of a security data lake leveraging ELK: A commercial solution is augmented to fill the gaps–enriching and highlighting security data by creating parsers for the data sources, integrations with other security products, health-check monitoring, and fine-tuning for performance.

Organizations that are looking for a security log management solution should evaluate the DIY ELK versus the cybersecurity vendors who process log data. Consider that a commercial security data lake can lower your total cost of ownership without volume-based pricing. A commercial security data lake will also provide tighter integrations to help your SOC team, will better highlight security data, and can provide analysts with greater visibility and search capabilities for better cyber threat detection.