Kubernetes is an open source platform to manage containerized applications. it enables you to manage, scale and deploy your containerized apps in a clustered environment. We can orchestrate our containers across various hosts with Kubernetes, scale the containerized apps with all resources on the fly, and have a centralized container management environment. In effect, while Kubernetes solves old problems, it can also create new ones. Specifically, adopting containers and container orchestration requires teams to rethink and adapt their monitoring strategies to account for the new infrastructure layers introduced in a distributed Kubernetes environment.
Centralized logging is an instrumental component of running and managing Kubernetes clusters at scale. Developers need access to logs for debugging and monitoring applications, operations teams need access for monitoring applications, and security needs access for monitoring. However, tasks such as log collection and metrics aggregation can be challenging within Kubernetes and Docker deployments due to the stateless natures of pods and containers. Open source tools like Elasticsearch, Kibana and Fluent Bit also known as EFK, provide effective methods to collect and store a wide variety of operational data to simplify these and similar problems.
In this webinar we will deploy an Elasticsearch cluster in Kubernetes and use Fluent bit to collect, transform, and ship log data to the Elasticsearch backend. Fluent Bit is an open source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. Its fully compatible with Docker and Kubernetes environments.
We will set up on our Kubernetes nodes to tail container log files, filter and transform the log data, and deliver it to the Elasticsearch cluster, where it will be indexed and stored. We will then use Kibana to Visualise the logs and create Interactive dashboards.
We will discuss:
Kubernetes monitoring Challenges
Why Opendistro for Elasticsearch for monitoring Kubernetes
Deploy Elasticsearch and Kibana on Kubernetes
Deploy Fluent Bit
With over 13 years of IT industry experience, Mussa specializes in the field of Big Data, Low Latency Monitoring, Network and Security Technologies, Network Solutions and has worked with large multinational corporations. He has a vast experience with open source technologies such as Elasticsearch and Kafka and has been involved in designing and deployment of data solutions for big financial organizations. Mussa is a big fan of open source technologies and likes getting involved in all kinds of data projects that involve open source technologies