AWS Log Analytics: Cloud Services and Reference Architecture

What is AWS Log Analytics? 

Log analysis is the process of collecting, parsing, and analyzing log data from various sources in order to identify patterns, trends, and issues. It is a critical tool for IT professionals, system administrators, and cybersecurity professionals to monitor and maintain the health, security, and performance of systems and applications.

In AWS, log analysis is important for a variety of reasons:

• Security: Log analysis can help identify security breaches or anomalies in system behavior that may indicate a security threat.
• Performance: Log analysis can help identify performance issues or bottlenecks in systems or applications, allowing IT professionals to troubleshoot and fix problems before they impact users.
• Compliance: Log analysis can help organizations meet regulatory and compliance requirements by providing a complete and accurate record of system activity.
• Debugging: Log analysis can help identify the root cause of errors or issues in systems or applications, allowing IT professionals to fix problems more quickly and efficiently.
• Cost optimization: Log analysis can help organizations optimize their AWS resources and reduce costs by identifying underutilized or overutilized resources.

This is part of a series of articles about log management.

---

AWS Services for Log Analytics

CloudWatch Logs Insights

CloudWatch Logs Insights is a fully managed service provided by AWS that allows users to search and analyze log data from their applications, resources, and services in real-time. It provides advanced query and visualization capabilities that enable users to identify trends, patterns, and issues in their log data.

With CloudWatch Logs Insights, users can:

• Run ad-hoc queries on their log data to identify trends, patterns, and issues.
• Visualize log data using charts and graphs.
• Set up alarms to be notified when specific events or patterns are detected in the log data.
• Integrate with other AWS services, such as Amazon Elasticsearch Service, to further analyze and visualize log data.

CloudWatch Logs Insights is a powerful tool for IT professionals, system administrators, and cybersecurity professionals to monitor and maintain the health, security, and performance of their systems and applications in AWS.

AWS CloudTrail

AWS CloudTrail is a service provided by AWS that enables users to track and monitor the activity in their AWS accounts. CloudTrail records API calls made in their accounts, including the identity of the API caller, the time of the API call, the source IP address of the API caller, the request parameters, and the response elements returned by the AWS service.

CloudTrail is an important tool for IT professionals, system administrators, and cybersecurity professionals to:

• Monitor and audit changes to their AWS resources.
• Identify and troubleshoot issues with their AWS resources.
• Meet compliance requirements by providing a complete and accurate record of AWS activity.
• Improve security by identifying potential threats or anomalies in their AWS accounts.

CloudTrail records are stored in an S3 bucket, and users can use CloudWatch Logs Insights, Amazon Elasticsearch Service, or other tools to analyze and visualize the log data.

Amazon OpenSearch Service

Amazon OpenSearch Service is a fully managed search and analytics service provided by AWS that is based on the open source Elasticsearch project. It enables users to search, analyze, and visualize data from a variety of sources, including websites, applications, and logs.

OpenSearch is designed to handle large amounts of data quickly and efficiently, and it offers a range of features for searching, analyzing, and visualizing data, including:

• Distributed search and analytics: OpenSearch is distributed across multiple servers and data centers, allowing users to scale their search and analytics capabilities as their data grows.
• Customizable search: OpenSearch allows users to customize the search experience by specifying the fields and filters they want to search, and by adjusting the relevance and ranking of search results.
• Trace analytics: OpenSearch includes a feature called “Trace Analytics” that allows users to analyze log data and identify trends, patterns, and issues in their systems and applications. This can be especially useful for IT professionals and cybersecurity professionals to monitor and maintain the health, security, and performance of their systems.

---

Centralized Logging Solution Architecture on AWS 

The Centralized Logging Solution on AWS is a design pattern that allows organizations to collect, process, and analyze log data from multiple sources in a centralized location. It typically includes the following components:

• Log ingestion: This refers to the process of collecting log data from various sources, such as applications, servers, and devices. This can be done using tools such as CloudWatch Logs, CloudTrail, or third-party log collectors.
• Log indexing: This refers to the process of storing and organizing log data in a searchable format. This can be done using tools such as Amazon Elasticsearch Service, which is based on the open source Elasticsearch project.
• Visualization: This refers to the process of presenting log data in a visual format, such as charts and graphs. This can be done using tools such as CloudWatch Logs Insights, Amazon Elasticsearch Service, or third-party visualization tools.

Deploying the Architecture Automatically with CloudFormation

The Centralized Logging Solution’s implementation guide is a set of instructions and resources provided by AWS for implementing a centralized logging solution using AWS services. It includes a detailed walkthrough of the steps involved in setting up a centralized logging solution, as well as accompanying AWS CloudFormation templates that automate the process of creating and configuring the necessary AWS resources.

The AWS CloudFormation templates included in the implementation guide are designed to streamline the process of setting up a centralized logging solution on AWS. They allow users to define the resources they need in a JSON or YAML file, and then use the AWS CloudFormation service to create and configure those resources automatically. This can save time and reduce the risk of errors compared to manually creating and configuring resources.

The implementation guide and accompanying CloudFormation templates are a useful resource for IT professionals, system administrators, and cybersecurity professionals who want to set up a centralized logging solution on AWS. They provide a step-by-step guide and automation tools to help users get started quickly and efficiently.

---

AWS Log Analytics with Exabeam UEBA

Managing cloud security can be a challenge, particularly as your data, resources and services grow. Misconfiguration and lack of visibility are frequently exploited in data and system breaches. Both issues are more likely to occur without centralized tools — and visibility is the key to good security monitoring. 

Provider dashboards and services may be enough to provide basic visibility. However, most organizations need more advanced security measures. This is why Log Analytics solutions are often combined with user and entity behavior analysis (UEBA) tools, like those found within Exabeam Fusion or Exabeam Security Investigation. Exabeam UEBA capabilities in these products create baselines of “normal” activity and can identify and alert to activity that deviates from the baseline. 

Exabeam brings in Log Analytics data and normalizes it with other session activity and signals related to the same user or entities on the network to:

• Providing centralized monitoring – dispersed systems can be a challenge to monitor as you may have individual dashboards and portals for each service. While Log Analytics can alert you to suspicious or policy-breaking behavior that you might otherwise miss in standalone dashboards, Exabeam Smart Timelines(™) let you see the activity in the context of the whole user session..
• Creating visibility in multi and hybrid cloud systems – cloud-specific services may not be extendable to on-premises resources and vice versa. Log Analytics can help you ensure that policies and configurations are consistent across environments. For example, by monitoring data use and transfer in hybrid storage services. Combined with a DLP solution or other local AD or AD FS signals, Exabeam helps you see anomalies as they happen.
• Helping you evaluate and prove compliance standards – Log Analytics can provide trackable, unified logging with evidence of actions taken. You can use Log Analytics logging and event tracking in compliance audits and certifications. Exabeam supports your compliance initiatives by providing Dashboards and audit trails for actions taken on the platform, as well as helping you store the logs affordably for the duration of your compliance needs. Exabeam offers APIs to export Search and query results into other systems as well, helping your entire security ecosystem operate as one.

Scaling to match your system needs – Log Analytics often use daemons or agents to monitor distributed systems. These agents allow you to scale your Log Analytics to match your environment size. You can take advantage of the scalability of any tools you use by accepting and incorporating data streams for tools across your system. Exabeam also scales with your needs, up to 100PB per tenant with processing log data at 1M+ EPS.

Exabeam offers Threat Detection, Investigation, and Response (TDIR) for Public Cloud as a cloud-specific content package that enables organizations to begin to understand “normal” within their cloud infrastructure and identify anomalous user behavior. This content is designed to help organizations combat cloud threats and aligned to the following Exabeam Use Cases:

• Compromised Credentials & Lateral Movement
• Data Exfiltration
• Privileged Activity
• Malware

This content is available for Exabeam Security Analytics, Exabeam Security Investigation, and Exabeam Fusion.