Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
Kubernetes provides three core components for container orchestration:
Control plane – The orchestration component responsible for the management of Kubernetes components such as clusters, nodes, and pods, and the networking among them.
Management plane – The administration layer, with the components that administrators use to interact with Kubernetes. Such components include the Kubernetes API, Custom Resource Definitions (CRDs), manifests, Helm charts, and so on.
Data plane – The traffic‑handling layer, with overlay and underlay networks among nodes, pods, and containers, as well as application services (apps and containers) which rely on the networking components. An application container in one pod communicating with an application container in another pod, using gRPC for example, is participating in the data plane.
Open-Source Kubernetes Monitoring Tools
1. Sematext
Sematext Monitoring is a monitoring solution for both traditional and microservice-based applications deployed on Kubernetes, capturing, metrics, and events in real time. You can then structure, visualize and analyze this data, set alerts on it, etc. Sematext Monitoring is part of Sematext Cloud, a cloud monitoring solution that takes care of both Kubernetes monitoring and logging without running any storage or monitoring infrastructure yourself. Sematext allows you to set up alerts on both logs and metrics, build log analytics reports and customizable monitoring dashboards. It making it much easier and faster to point out problematic pods than using traditional monitoring or command-line tools. When alerts are triggered you get notified through email, Slack, or any other notification hook of your choosing.
Pros:
Easy to install
Auto-discovery finds services and logs and monitor them without installing anything
Built-in alerting and anomaly detection
Default monitoring dashboards and alert rules to save time
Offered as SaaS, no infrastructure to manage
2. Kubernetes Dashboard
Kubernetes Dashboard is a web-based UI add-on for Kubernetes clusters. It provides a simple way to manage, troubleshoot and monitor your environment. You can use the Kubernetes Dashboard to see basic metrics related to memory and CPU usage statistics across all of your nodes and to monitor the health of workloads (pods, deployments, replica sets, cron jobs, etc.)
Pros:
Easy to install
Part of the Kubernetes ecosystem
Cons:
Limited features
3. Prometheus
Prometheus is one of the most popular open-source tools used to monitor Kubernetes. It was developed by SoundCloud and donated to the CNCF (Cloud Native Computing Foundation).
What’s different about Prometheus compared to other time-series databases – such as Cassandra, Graphite, InfluxDB – is that it has a simple yet powerful multidimensional data model and its flexible query language (PromQL). Furthermore, it follows a pull model rather than push and has built-in real-time alerting mechanisms. Finally, being open-source, it gathered a large community interested in helping and bringing innovation. Prometheus has no built-in dashboard, so you’ll need to use a separate visualization tool. Users opt for Grafana.
Pros:
Built-in monitoring and alerting
Functional and reliable during outages
Kubernetes-native, easy to use
Integrates well with Grafana
Large community
Cons:
No built-in long-term storage
No dashboards
No authentication/authorization
No anomaly detection
Doesn’t handle logs or traces, only metrics
Challenges at scale
4. Grafana
Grafana is an open-source solution used for monitoring, metrics, data visualization, and analysis.
Compared to other visualization tools, Grafana stands out because it connects with a long list of databases. When used to monitor Kubernetes, Grafana usually sits on top of Prometheus, but it’s also popular in combination with InfluxDB or Graphite. You can build comprehensive monitoring dashboards with a wide variety of graphs, from heatmaps to line graphs, bar graphs, histograms or Geo maps. And, you can already find a lot of ready-to-use Kubernetes monitoring dashboards. Grafana also features a built-in alerting system, along with filtering capabilities, annotations, data-source specific querying, authentication and authorization, cross-organizational collaboration, and many more. Grafana is easy to set up and use. It’s popular in the Kubernetes community and some deployment configuration files include a Grafana container by default.
Pros:
Includes support for Elasticsearch and Prometheus
Broad compatibility with various data sources
Great reporting and visualization functions
Active developer community
Alerting capabilities
Can query several entities at a time
Cons:
Not customized for Kubernetes log management
5. Jaeger
Jaeger is a free tracing tool used for monitoring and troubleshooting in complex distributed systems, including Kubernetes environments. It was released by Uber Technologies and open-sourced in 2016.
With Jaeger, users can perform root cause analysis, distributed transaction monitoring, distributed context propagations, service dependency analysis, and performance and latency optimization. Jaeger features OpenTelemetry-based support for Java, Node, Python, Go, and C++ and for various data sources, including Cassandra, Elasticsearch, Kafka, and memory.
Pros:
Various instrumentation options
Easy to deploy
Modern user interface
Cons:
Limited backend integration
6. Elastic Stack (ELK)
The ELK stack is among the most popular open-source log management solutions, including for Kubernetes. But it can easily be used – and it is used by many – for monitoring purposes too.
It’s a collection of four tools that ensures an end-to-end logging pipeline. Elasticsearch is a full-text search and analytics engine where you can store Kubernetes logs. Logstash is a log aggregator that captures and processes logs before shipping them to Elasticsearch. Kibana provides reporting and visualization functionalities. And finally, Beats are lightweight data shippers used to send logs and metrics to Elasticsearch. ELK comes equipped with Kubernetes and Docker monitoring beats with auto-discovery. The Beats collect Kubernetes and Docker logs, metrics and metadata, thus helping you monitor performance at application and system level.
Pros:
Rich analytics capabilities
Easy to deploy and run in Kubernetes environment
Large community
Cons:
Operating at scale requires a lot of expertise
7. cAdvisor
Container Advisor, officially called cAdvisor, gives you insight and understanding into the resource usage and performance characteristics of running containers. It is a running daemon that collects, aggregates, processes, and exports information about running containers. It keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics for each running container. This data is exported both on the container and machine-level.
For Kubernetes users, cAdvisor can be run as a DaemonSet.
Pros:
cAdvisor is an open source container resource usage collector.
Native support for Docker containers with support for other container types.
cAdvisor operates per node. It auto-discovers all containers in the given node and collects CPU, memory, filesystem, and network usage statistics.
Supports exporting stats to various storage plugins like Elasticsearch, InfluxDB etc.
Exposes raw and processed stats via a versioned remote REST API
Cons:
Collects only basic resource utilization
Doesn’t offer any long term storage, trending, or analysis capabilities.
Captures only metrics, not logs, traces, or events
8. Kubewatch
Kubewatch is a Kubernetes watcher that publishes notification to available collaboration hubs/notification channels. You run it in your Kubernetes cluster, and get event notifications through webhooks.
Once the Kubewatch pod is running, you will start seeing Kubernetes events in your configured Slack channel or any other webhook you configured.
Pros:
Simple setup
Instant notifications to preferred locations
Cons:
It only watches basic events
Doesn’t offer any long term storage, trending, or analysis capabilities.
9. Kube-state-metrics
The Kubernetes API server exposes data about the count, health, and availability of pods, nodes, and other Kubernetes objects. The kube-state-metrics add-on makes it easier to consume these metrics and help surface issues with cluster infrastructure, resource constraints, or pod scheduling. How does kube-state-metrics work? It listens to the Kubernetes API and generates metrics about the state of Kubernetes objects. These include node status, node capacity like CPU and memory, number of desired/available/unavailable/updated replicas per Deployment, different pod statuses, e.g., waiting, running, ready, and so on. Once you deploy kube-state-metrics to your cluster, it provides a vast array of metrics in text format on an HTTP endpoint. These metrics can be easily consumed by any monitoring system that can collect Prometheus metrics.
Pros:
Simple setup
Compatible with Prometheus
Cons:
It only watches basic Kubernetes API metrics
Doesn’t offer any long term storage, trending, or analysis capabilities.
10. Datadog
Datadog is an APM solution that enables you to extract logs, metrics, events and service states from Kubernetes in real time. It enables you to monitor, troubleshoot and optimize application performance.
Datadog features dashboards and high-resolution metrics and events for manipulation and graphing. You can also set up alerts and receive notifications on various channels, including Slack and PagerDuty.
The Datadog Agent is easy to install. You can run it using a DaemonSet that will be deployed to every cluster node. With the Datadog Agent successfully deployed, resource metrics and events will start coming into Datadog. You can view the data in the built-in Kubernetes dashboard that Datadog provides.
Pros:
Easy to install
Great APM integration
Cons:
Confusing logs integrations
Limited plans (only 1)
Expensive
11. New Relic
New Relic is a monitoring tool that features Kubernetes integration, giving you an overview of your servers, hosts, applications, and services. You can capture data and metadata for nodes, pods, containers, deployments, replica sets, and namespaces. It also features powerful searching capabilities, as well as tag-driven alerting and dashboarding. The New Relic Kubernetes integration monitors and tracks aggregated core and memory usage across all nodes in your cluster. This allows you to meet resource requirements for optimal application performance.
Pros:
Cluster health visualization
Integration with APM
Cons:
Confusing onboarding
Expensive, pricing by both usage and the number of users
UI feels very outdated
12. Sensu
Sensu offers an end-to-end observability pipeline where you can collect, filter, and transform monitoring events and send them to the database of your choosing. Sensu can be run in your Kubernetes cluster side-by-side with Prometheus to get the most out of both solutions. You can also run it natively without Prometheus.
Pros:
Simple setup
Compatible with Prometheus
Cluster health visualization
Cons:
Very basic GUI interface
Weak out-of-the-box dashboards
13. Dynatrace
Dynatrace monitors native Kubernetes and managed services like OpenShift, EKS, AKS, GKE, IBM IKS, etc. Your cloud platform, container runtime or service mesh layer does not matter, neither does it matter if you are running thousands of nodes or just a few. Dynatrace makes monitoring your Kubernetes infrastructure and workloads simple. By deploying and managing the Dynatrace OneAgent Operator, the OneAgent becomes a native, first-class citizen of the platform. It will track the availability, health, and resource utilization of your Kubernetes cluster.
Pros:
Easy to install
Great APM integration
Cons:
Steep learning curve
Expensive
Open-source Kubernetes Security Tools
1: Open Policy Agent (OPA)
OPA is a general-purpose policy engine, it is a very powerful tool for enforcing context-aware security policies. With the deprecation of Pod Security Policy initiated as of Kubernetes v.1.21 (and complete removal by v.1.25), many organizations will likely turn to OPA to fill in that gap.
2: KubeLinter
KubeLinter is a static analysis tool that scans YAML files and Helm charts. KubeLinter analyzes Kubernetes YAML files and Helm charts and checks them against a variety of best practices, with a focus on production readiness and security.
KubeLinter ships with default checks, designed to give you useful information about your Kubernetes YAML files and Helm charts. This helps teams check early and often for security misconfigurations and DevOps best practices. Some common examples of these include running containers as a non-root user, enforcing least privilege, and storing sensitive information only in secrets.
3: Kube-bench
Nearly a quarter of respondents use Kube-bench, a tool that audits Kubernetes settings against security checks recommended in the CIS Benchmark for Kubernetes. The scans are configured using YAML files, and the tool itself is written in Go, a familiar language to Kubernetes developers.
This tool is particularly useful when self-managing the control plane components.
4: Kube-hunter
Built by the same team behind Kube-bench, Kube-hunter looks for exploitable security weakness in Kubernetes clusters. One of the more useful features of Kube-hunter is the ability to exploit the vulnerabilities it discovers to look for further exploits. 23% of respondents use Kube-hunter.
5: Terrascan
Built on top of OPA, Terracan is an open source static code analyzer for Infrastructure as Code that is used by 22% of respondents. With over 500+ Policies for security best practices across various applications, including Terraform, Kubernetes (JSON/YAML), AWS, Azure, GCP, Kubernetes, and GitHub, Terrascan can detect security vulnerabilities and compliance violations and mitigate risks before provisioning infrastructure.
6: Falco
The only open source tool in this list that is built for runtime security, Falco is used by 21% of respondents to protect running containerized applications in Kubernetes. Falco also provides security policies that use contextual data from Kubernetes and kernel events to detect anomalous application behavior indicative of a threat.
7: Clair
Clair is an open source security tool used for scanning container images for known vulnerabilities. Clair is a static analysis tool, so it will not be able to detect vulnerabilities at runtime. Clair is used by 11% of respondents.
8: Checkov
Checkov is a static code analyzer for Infrastructure as code that is used by 9% of respondents. The latest version of Chekov introduced context-based analysis. It detects misconfigurations using graph-based scanning of cloud infrastructure that is provisioned with applications such as Terraform, Terraform plan, Cloudformation, Kubernetes, Dockerfile, Serverless, or ARM Templates.
Read More Articles on kubernetes:
Resource: Sematext.com, Wikipedia, Kubernetes
The Tech Platform