Kubernetes Security Monitoring: How to Gain Visibility and Prevent Breaches

Security monitoring is one of the most effective ways to discover threats and protect applications. However, it is also one of the biggest challenges. Kubernetes monitoring can help you improve security by actively monitoring clusters, containers and resource allocation of namespaces. This article explains the basics of Kubernetes and reviews the security challenges of Kubernetes deployment.

What is Kubernetes?

Kubernetes is a container orchestration system developed by Google. Kubernetes was released to the public in 2015 to answer the need of container management at a large scale. Today, developers use Kubernetes to scale, manage, and automate the deployment of containerized applications and services. 

How Kubernetes works

Kubernetes orchestrates containers by following a client-server architecture. The architecture utilizes many different components to control nodes and point of contact. 

The following basic components are useful to know as you start using Kubernetes:

• Containers—a standard unit of software that packages and isolates applications with their entire runtime environment and necessary files.
• Pods—a group of internally connected containers. Containers inside a pod share the same storage volume, IP address, and resources.
• Controllers—control and provide visibility of Kubernetes tasks. 
• Ingress—a set of rules that defines the communication process of Kubernetes with external apps and services.
• Service—a collection of pods that run on the same cluster. A cluster can have as many services as you need. You can use services to run a microservice architecture.
• Deployment—the goal of a Kubernetes deployment is to run multiple replicas of the application. Deployment  automatically replaces unresponsive or failed instances, based on the pod template specs.
• ConfigMap—used to push spec information from Kubernetes into the containers, without stopping running processes.
• Pod templates—defines the configuration of each pod in the deployment. This includes the job type of each container, and the label and volume of a pod.
• ReplicaSet—an application programming interface (API) object that manages pod scaling. ReplicaSet enables fast replication of pods.

Kubernetes features

Kubernetes automates container processes, like rollouts and rollbacks, cluster deployment, and bin packing. Kubernetes is flexible, and scalable. You can use Kubernetes in multiple environments, including cloud-native ecosystems and on-premise data centers. You can also balance your loads by monitoring services.

Kubernetes security challenges

Microservice architecture is the biggest security challenge of Kubernetes. Every container you deploy opens a new point of entry for attackers. As a result, your attack surface increases. This makes Kubernetes an easy target, since Kubernetes automates most container management tasks. 

Regulatory compliance

Moreover, the microservice architecture complicates the regulatory compliance of Kubernetes. The official Kubernetes documents explain how to secure Kubernetes configurations, and comply with regulations. However, DevOps and DevSecOps teams are responsible for implementing those instructions and enforce continuous compliance using automation.

Unauthorized access  

Kubernetes manage containers on the host with container runtime orchestration tools. As a result, attackers can access your system during runtime, and deploy unauthorized containers.

Malicious code

Attackers can compromise containers to run malicious code. Kubernetes cannot automatically protect against runtime attacks. You need to actively scan for vulnerabilities throughout the entire container lifecycle. Active vulnerability scanning is achieved through integration of security from the very start. 

Examples of Kubernetes Attacks

Kubernetes attacks can originate on external networks or by malicious insiders. Here are a few examples:

• Container compromise—application vulnerabilities or misconfiguration enables attackers to search for weaknesses in the system, network, files, or process controls.
• Unauthorized pod connections—compromised containers can connect with other running pods on the same or other hosts to launch an attack. You can detect these attacks only with layer 7 network filtering.
• Data loss—attackers use techniques like reverse shell in a pod and network tunneling to hide confidential data.

What to Monitor in Kubernetes?

There are many metrics you can monitor to ensure the security of Kubernetes. Typically, they are divided into three main categories, infrastructure monitoring, services monitoring, and internal monitoring.

Infrastructure

You have to monitor all the underlying server components of a cluster because issues at the server level will show up in the workloads.

CPU monitoring

Monitoring the CPU will reveal both user and system consumption. In addition, CPU monitoring will show the iowait metric. Iowait indicates bottlenecks in the storage read and write processes when running clusters in the cloud or with any network storage. A storage framework with limited capacity can impact performance.

Pod monitoring

The Kubernetes scheduler ensures that pods are assigned to nodes with available resources. Therefore, you should consider how many nodes can fail before the other nodes can no longer handle all the necessary resources. 

Make sure that the remaining nodes can handle the workload during the time it takes to bring the failed node back online. Another option is to use a service like cloud autoscaling group that accelerates recovery.

Disk space

Running out of disk space can be disastrous when you run write-intensive services like etcd or datastore. The lack of disk space can result in real damage. That damage can lead to real-world business losses. You must monitor your available disk space with technologies like logical volume manager (LVM).

Internal Services

Kubernetes enables you to monitor the applications directly by exposing internal resource metrics. Kubernetes usually can maintain the desired state of an application. However, sometimes you need to fix the issue manually.

Kubernetes Services

You need to monitor the components that are critical to the health of your applications. This includes the Kubernetes master and worker nod components like etcd. The monitoring tool needs to detect the failure and either fix it or send an alert if any of these critical components fails.

Top security monitoring tools for Kubernetes

The list below reviews some of the most popular security monitoring tools for Kubernetes.

Prometheus

Prometheus is an open source monitoring and alerting tool. Prometheus is based on an HTTP pull model to record real-time metrics in a time series database. 

Key features include:

• Kubernetes integration—Kubernetes uses Prometheus as the default monitoring system. The integration supports service discovery and dynamic scheduling of services.
• Built-in alert manager—enables you to send notifications according to specified rules and channels. As a built-in feature, the alert manager enables you to avoid the use of external systems or APIs. 
• Multi-dimensional data model—provides a label-based, time-series database that you can query with the PromQL language. 

Prometheus can also work with additional tools, which you can add to extend the default monitoring features offered by Kubernetes. For example, this guide explains how to use Kubernetes Prometheus with Cortex. Or you can use the two tools reviewed below.

Grafana

Grafana is an open-source platform for data monitoring, visualization, and analysis. You can create dashboards with different panels. Each panel represents particular metrics over a set time-frame. 

Grafana supports table, graph, heatmap, singlestat, and freetext panels. In addition, you can integrate Grafana with official and community-built plugins like clock and world map.

Key features include:

• Annotations—shows a graph marker in Grafana. You can use annotations for data correlation in case of a failure. You can create the annotations manually by clicking on a graph and input some text. You can also upload data from any data source.
• Dashboard templating—templating enables you to create dashboards that you can reuse for multiple use cases. For example, you can use the same dashboard for a test server or a production server.
• Custom plugins—enables you to extend Grafana with third-party integrations. Some popular integrations include Zabbix, the Worldmap Panel for data visualization, and Influx Admin Panel.
• Provisioning—you can use scripts to automate Grafanas. For instance, you can automatically set up monitoring for a new Kubernetes cluster with a script that contains the server, IP address, and data sources preset.

cAdvisor

cAdvisor enables developers to understand the performance characteristics and resource usage of their containers. cAdvisor is a running daemon that gathers, stores, processes, and exports information about running containers.

Key features include:

• Auto-discovery—automatically discovers all containers in a given node and collects statistics like memory, CPU, network, and filesystem usage.
• Storage plugins—exports stats to different storage plugins like Elasticsearch, and InfluxDB.
• CPU monitoring—monitors metrics like time duration the container processes run on the CPU, and value of container cpu average load.

Conclusion

Kubernetes makes your applications faster. However, that does not eliminate the need to keep an eye on security. The responsible thing to do is to deploy a monitoring system that keeps you informed and helps you to make data-driven decisions.

Tools like Prometheus can monitor Kubernetes metrics you can use for security. Tools like Grafana display metrics on multiple dashboards to make sure you are not missing any security threat.