Kafka监控

# Kafka 监控

# Kafka Eagle

先关闭所有 kafka

修改 kafka 启动命令

cd /opt/module/kafka/bin
vim kafka-server-start.sh

1
2

修改以下内容

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
    #export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

1
2
3
4
5

xsync /opt/module/kafka/bin/kafka-server-start.sh

再启动 kafka

安装 eagle

tar -zxvf kafka-eagle-bin-1.3.7.tar.gz
cd kafka-eagle-bin-1.3.7
tar -zxvf kafka-eagle-web-1.3.7-bin.tar.gz -C /opt/module/
cd /opt/module/
mv kafka-eagle-web-1.3.7/ eagle
cd eagle/bin/
chmod +x ke.sh
vim ke.sh

1
2
3
4
5
6
7
8

修改 ke.sh

######################################
# multi zookeeper&kafka cluster list
######################################
kafka.eagle.zk.cluster.alias=cluster1
cluster1.zk.list=hadoop102:2181,hadoop103:2181,hadoop104:2181

######################################
# kafka offset storage
######################################
cluster1.kafka.eagle.offset.storage=kafka

######################################
# enable kafka metrics
######################################
kafka.eagle.metrics.charts=true
kafka.eagle.sql.fix.error=false

######################################
# kafka jdbc driver address
######################################
kafka.eagle.driver=com.mysql.jdbc.Driver
kafka.eagle.url=jdbc:mysql://hadoop102:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=000000

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

添加环境变量

vim /etc/profile.d/my_env.sh

export KE_HOME=/opt/module/eagle
export PATH=$PATH:$KE_HOME/bin

1
2

source /etc/profile.d/my_env.sh

启动

ke.sh start

访问 http://192.168.130.102:8048/ke 用户 admin 密码 123456

# Kafka Monitor

目前已经停止更新了不推荐使用

https://github.com/Morningstar/kafka-offset-monitor/releases

mkdir -p /opt/module/kafka-offset-console
cd /opt/module/kafka-offset-console

1
2

将 KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar 上传到集群中的 /opt/module/kafka-offset-console

创建脚本文件 start.sh

vim start.sh

#!/bin/bash
nohup java -cp KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--offsetStorage kafka \
--kafkaBrokers hadoop102:9092,hadoop103:9092,hadoop104:9092 \
--kafkaSecurityProtocol PLAINTEXT \
--zk hadoop102:9092,hadoop103:9092,hadoop104:9092 \
--port 8086 \
--refresh 10.seconds \
--retain 2.days \
--dbName offsetapp_kafka >/dev/null 2>&1 &

1
2
3
4
5
6
7
8
9
10
11

chmod +x start.sh

访问 http://hadoop102:8086 / 查看 web

# Kafka Manager(CMAK)

https://github.com/yahoo/CMAK

CMAK v3.0.0.5 需要 JDK11 以上，如果你想用 JDK1.8 去支持此版本是行不通的，会报错提示找不到有些 class 文件

yum -y install java-11-openjdk-devel.x86_64

修改配置文件

suod vim /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el7_9.x86_64
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

1
2
3

source /etc/profile
java -version

1
2

上传解压

mkdir -p /opt/module/kafkamanager
cd /opt/module/kafkamanager

1
2

修改配置

vim conf/application.conf

kafka-manager.zkhosts="hadoop102:2181,hadoop103:2181,hadoop104:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="hadoop102:2181,hadoop103:2181,hadoop104:2181"
cmak.zkhosts=${?ZK_HOSTS}

1
2
3
4

启动

chmod +x bin/cmak
bin/cmak

1
2

访问 http://hadoop102:9000/

编辑

上次更新: 2023/12/06, 01:31:48

← Flume 对接 Kafka Kafka面试题→