What is SIEM? Complete Guide to the Future SOC
01

What is SIEM?

Security Information and Event Management (SIEM) solutions use rules and statistical correlations to turn log entries, and events from security systems, into actionable information. This information can help security teams detect threats in real time, manage incident response, perform forensic investigation on past security incidents, and prepare audits for compliance purposes.

 

The term SIEM was coined in 2005 by Mark Nicolett and Amrit Williams, in Gartner’s SIEM report, Improve IT Security with Vulnerability Management. They proposed a new security information system, on the basis of two previous generations. 

  • Security Information Management (SIM) – a first generation, built on top of traditional log collection and management systems. SIM introduced long-term storage, analysis, and reporting on log data, and combined logs with threat intelligence.

  • Security Event Management (SEM) – a second generation, addressing security events – aggregation, correlation and notification for events from security systems such as antivirus, firewalls and Intrusion Detection Systems (IDS), as well as events reported directly by authentication, SNMP traps, servers, databases etc.

In the years that followed, vendors introduced systems that provided both security log management and analysis (SIM) and event management (SEM), to create Security Information Event Management (SIEM) solutions.

SIEM security platforms can aggregate both historical log data and real-time events, and establish relationships that can help security staff identify anomalies, vulnerabilities and incidents.

The main focus is on security-related incidents and events, such as succeeded or failed logins, malware activities or escalation of privileges.

These insights can be sent as notifications or alerts, or discovered by security analysts using the SIEM platform’s visualization and dashboarding tools.

Next Gen SIEM

SIEM is a mature technology, and the next generation of SIEMs provide new capabilities:

  • User Event Behavioral Analysis (UEBA) advanced SIEMs go beyond rules and correlations, leveraging AI and deep learning techniques to look at patterns of human behavior. This can help detect insider threats, targeted attacks, and fraud.

  • Security Orchestration and Automation (SOAR) – next-gen SIEMs integrate with enterprise systems and automate incident response. For example, the SIEM might detect an alert for ransomware and perform containment steps automatically on affected systems, before the attacker can encrypt the data.

  • Want to learn more about SIEM?
    Have a look at these articles:

What Can a SIEM Help With?

Components and Capabilities
  • Data aggregation

    Data aggregation

    Aggregates data from network, security, servers, databases, applications, and other  security systems like firewalls, anti virus and Intrusion Detection Systems (IDS)

  • Threat intelligence feeds

    Threat intelligence feeds

    Combines internal data with threat intelligence feeds containing data on vulnerabilities, threat actors and attack patterns

  • Correlation

    Correlation

    Links events and related data into meaningful bundles which represent a real security incident, threat, vulnerability or forensic finding

  • Analytics

    Analytics

    Uses statistical models and machine learning to identify deeper relationships between data elements, and anomalies compared to known trends, and tie them to security concerns

  • Alerting

    Alerting

    Analyses events and sends out alerts to notify security staff of immediate issues, either by email, other types of messaging, or via security dashboards

  • Dashboards and visualizations

    Dashboards and visualizations

    Creates visualizations to allow staff to review event data, see patterns and identify activity that does not conform to standard patterns

  • Compliance

    Compliance

    Automates the gathering of compliance data, producing reports that adapt to security, governance and auditing processes for standards like HIPAA, PCI/DSS, HITECH, SOX and GDPR

  • Retention

    Retention

    Stores long-term historical data to enable analysis, tracking, and data for compliance requirements. Especially important in forensic investigations, which happen after the fact

  • Threat hunting

    Threat hunting

    Allows security staff to run queries on SIEM data, filter and pivot the data, to proactively uncover threats or vulnerabilities

  • Incident Response

    Incident Response

    Provides case management, collaboration and knowledge sharing around security incidents, allowing security teams to quickly synchronize on the essential data and respond to a threat

  • SOC Automation

    SOC Automation

    Integrates with other security solutions using APIs, and lets security staff define automated playbooks and workflows that should be executed in response to specific incidents

How SIEM Works

Present and Future

In the past, SIEMs required meticulous management at every stage of the data pipeline - data ingestion, policies, reviewing alerts and analyzing anomalies. Increasingly, SIEMs are getting smarter at pulling data together, from ever more organizational sources, and using AI techniques to understand what type of behavior constitutes a security incident.

  1. Data Collection 01

    Data Collection

    Most SIEM systems collect data by deploying collection agents on end-user devices, servers, network equipment, or other security systems like firewalls and antivirus, or via protocols syslog forwarding, SNMP or WMI. Advanced SIEMs can integrate with cloud services to obtain log data about cloud-deployed infrastructure or SaaS applications, and can easily ingest other non-standard data sources.

    Pre-processing may happen at edge collectors, with only some of the events and event data passed to centralized storage.

  2. Data Storage 02

    Data Storage

    Traditionally, SIEMs relied on storage deployed in the data center, which made it difficult to store and manage large data volumes.

    As a result, only some log data was retained. Next-generation SIEMs are built on top of modern data lake technology such as Amazon S3 or Hadoop, allowing nearly unlimited scalability of storage at low cost. This makes it possible to retain and analyze 100% of log data across even more platforms and systems.

  3. Policies and Rules 03

    Policies and Rules

    The SIEM allows security staff to define profiles, specifying how enterprise systems behave under normal conditions.

    They can then set rules and thresholds to define what type of anomaly is considered a security incident. Increasingly, SIEMs leverage machine learning and automated behavioral profiling to automatically detect anomalies, and autonomously define rules on the data, to discover security events that require investigation.

  4. Data Consolidation and Correlation 04

    Data Consolidation and Correlation

    The central purpose of a SIEM is to pull together all the data and allow correlation of logs and events across all organizational systems.

    An error message on a server can be correlated with a connection blocked on a firewall, and a wrong password attempted on an enterprise portal. Multiple data points are combined into meaningful security events, and delivered to analysts by notifications or dashboards. Next-gen SIEMs are getting better and better at learning what is a “real” security event that warrants attention.

What are SIEMs Used For

  • Security Monitoring 01

    Security Monitoring

    SIEMs help with real-time monitoring of organizational systems for security incidents.

    A SIEM has a unique perspective on security incidents, because it has access to multiple data sources – for example, it can combine alerts from an IDS with information from an antivirus product. It helps security teams identify security incidents that no individual security tool can see, and help them focus on alerts from security tools that have special significance.

  • Advanced Threat Detection 02

    Advanced Threat Detection

    SIEMs can help detect, mitigate and prevent advanced threats, including:

    • Malicious insiders – a SIEM can use browser forensics, network data, authentication and other data to identify insiders planning or carrying out an attack
    • Data exfiltration (sensitive data illicitly transferred outside the organization) – a SIEM can pick up data transfers that are abnormal in their size, frequency or payload
    • Outside entities, including Advanced Persistent Threats (APTs) – a SIEM can detect early warning signals indicating that an outside entity is carrying out a focused attack or long-term campaign against the organization
  • Forensics and Incident Response 03

    Forensics and Incident Response

    SIEMs can help security analysts realize that a security incident is taking place, triage the event and define immediate steps for remediation.

    Even if an incident is known to security staff, it takes time to collect data to fully understand the attack and stop it – SIEM can automatically collect this data and significantly reduce response time. When security staff discover a historic breach or security incident that needs to be investigated, SIEMs provide rich forensic data to help uncover the kill chain, threat actors and mitigation.

  • Compliance Reporting and Auditing 04

    Compliance Reporting and Auditing

    SIEMs can help organizations prove to auditors and regulators that they have the proper safeguards in place and that security incidents are known and contained.

    Many early adopters of SIEMs used it for this purpose – aggregating log data from across the organization and presenting it in audit-ready format. Modern SIEMs automatically provide the monitoring and reporting necessary to meet standards like HIPAA, PCI/DSS, SOX, FERPA and HITECH.

SIEM Best Practices

The Infosec Institute suggests 10 best practices for successful implementation of a SIEM platform.

Defining SIEM requirements:

  • Define requirements for monitoring, reporting and auditing, consulting all relevant stakeholders before deploying a SIEM.

  • Determine the scope of the SIEM – which parts of the infrastructure it will cover, necessary credentials, and log verbosity.

  • Define audit data accessibility, retention, how to achieve data integrity, evidentiary rules, and disposal for historical or private data.

Ensure you leverage the SIEM to monitor and report on all of the following:

  • Access monitoring – transgression and anomalous access to key resources

  • Perimeter defenses – status of perimeter defenses, possible attacks and risky configuration changes

  • Resource integrity – critical network resources – status, backups, change management, threats and vulnerabilities

  • Intrusion detection – incidents reported by intrusion detection, or correlated/inferred using SIEM data

  • Malware defense – violations, threats, or activity regarding malware controls

  • Application defenses – status, configuration changes, violations and anomalies for web servers, databases and other web app resources

  • Acceptable use – status, issues and violations regarding acceptable, mandated or metered use of system resources

SIEM Evolution

2005
Generation I

Early SIEM

Early SIEM

The first SIEMs combined Security Information Management (SIM) and Security Event Management (SEM). They were limited in scale of data managed and supported alerting/visualizations.

  • Scalability Scales vertically
  • Historic Data Partial
  • Data Collection Slow manual ingestion of log data
  • Threat Detection Manual analysis and alerts based on manual rules
  • Incident Response Little or no interface with downstream systems
  • Dashboards And Visualizations Very limited
2010
Generation II

Big Data SIEM

Big Data SIEM

An integrated SIEM based on big data infrastructure, managing and correlating historical log data, real-time events and threat intelligence in one place – providing a holistic view of enterprise security data.

  • Scalability Scales horizontally, supporting big data
  • Historic Data Full, with some filtering
  • Data Collection Automated ingestion, data sources limited
  • Threat Detection Manual analysis, alerts and dashboards
  • Incident Response Limited interface with downstream systems
  • Dashboards And Visualizations Typically limited set of pre-built visualizations
2017
Generation III

Automation and Machine Learning

Automation and Machine Learning

Early SIEMs had limited ability to proactively warn about and react to complex security events. New SIEMs perform automated behavioral profiling (UEBA), and can automatically interact with IT and security systems to mitigate incidents (SOAR).

  • Scalability Based on data lake, unlimited scale
  • Historic Data Unlimited historic retention including new data sources like the cloud
  • Data Collection Automated ingestion of any data source
  • Threat Detection Automated, based on machine learning and behavioral profiling
  • Incident Response Integrates with IT and security tools, full Security Orchestration and Automation (SOAR) capabilities
  • Dashboards And Visualizations Full BI data exploration

Next-Generation SIEMs

The Future is Here

Integrates with IT and security tools, full Security Orchestration and Automation (SOAR) capabilities

New SIEM platforms provide advanced capabilities such as:
  • Complex threat identification – correlation rules can’t capture many complex attacks, because they lack context, or can’t respond to new types of incidents. With automatic behavioral profiling, SIEMs can detect behavior that suggests a threat.
  • Lateral movement – attackers move through a network by using IP addresses, credentials and machines, in search of key assets. By analyzing data from across the network and multiple system resources, SIEMs can detect this lateral movement.
  • Entity behavior analysis – critical assets on the network such as servers, medical equipment or machinery have unique behavioral patterns. SIEMs can learn these patterns and automatically discover anomalies that suggest a threat.
  • Detection without rules or signatures – many threats facing your network can’t be captured with manually-defined rules or known attack signatures. SIEMs can use machine learning to detect incidents without pre-existing definitions.
  • Automated incident response – once a SIEM detects a certain type of security event, it can execute a pre-planned sequence of actions to contain and mitigate the incident. SIEMs are becoming full Security Orchestration and Automation (SOAR) tools.
  • An example of a next-generation SIEM is the Exabeam Security Management Platform (SMP), which combines behavioral analytics based on machine learning, cloud connectors, a flexible data lake infrastructure, incident response and threat hunting capabilities.
Prev What is SIEM Next Introduction

CH04

UEBA

User and Entity Behavioral Analytics detects threats other tools can’t see

Read More