Flume 对接 Kafka
# Flume 对接 Kafka
Flume 输出到 Kafka 中
Flume 配置文件
a2.sources = r2
a2.sinks = k2
a2.channels = c2
a2.sources.r2.type = exec
a2.sources.r2.command = tail -F /opt/module/flume/datas/123.log
a2.sources.r2.shell = /bin/bash -C
#kafka Sink
a2.sinks.k2.type = org.apache.flume.source.kafka.KafkaSource
#发送topic
a2.sinks.k2.kafka.topic = first
#kafka地址
a2.sinks.k2.kafka.bootstrap.servers = hadoop102:9092,hadoop103:9092,hadoop104:9092
a2.sinks.k2.kafka.flumeBatchSize = 20
#ack模式
a2.sinks.k2.kafka.producer.acks = all
a2.sinks.k2.kafka.producer.linger.ms = 1
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
启动 flume
flume-ng agent -n a2 -c conf/ -f datas/flume-taildir-hdfs.conf -Dflume.root.logger=INFO,console
1
同时 flume 给我们提供了 kafka source 可以从 kafka 指定话题中读取数据到 channel 中
编辑 (opens new window)
上次更新: 2023/12/06, 01:31:48