Services: Operations & Monitoring Kafka

Operations & Monitoring Kafka

We suppose that you have already integrated Kafka into your IT system and it is making an impact on your data processing. Congratulations! You have already taken a significant step towards real-time.

Kafka is the most reliable and scalable streaming platform to move large amounts of data quickly. However, you need a little extra to achieve the maximum efficiency of your data architecture.

The standard Kafka does not provide all functions to look inside the data pipes for day to day operations. Our Kafka monitoring tool makes easier to monitor Kafka Eco System and track data flows between producer service and consumer service in real-time and the system also ensures that alerts are sent to you.

The Kafka monitoring tool sends an alert when:

Kafka cluster or any service is down.
Application is lagging to consume messages.
Unconsumed messages are gradually growing.
JVM memory utilisation goes above threshold.

Other Advantages and Benefits

Monitors critical low latency application where produced data should be consumed immediately
In case of alerts, notifies on multiple channels like email/slack/SMS etc.
Can be integrated with multiple applications.

Technical Insight

This solution will produce metrics data for regular interval and can be produced to support Kafka monitoring Prometheus and Grafana monitoring stack.

This use case can be enhanced to include other monitoring metrics like service monitoring, log monitoring, application performance monitoring and similar.

Looking inside Kafka requires you to be able to monitor how many messages are published and how many of them are consumed by each consumer group.

In the Messages position diagram, the total messages produced till offset are 50. However, only 30 messages are consumed till the offset.

The difference between produced messages and consumed messages is 20 and so, the consumer is lagging.

This solution connects with Kafka and monitors current producer offset position and current consumer offset position. It sends alerts when the difference between consumer and producer offset keeps increasing.