`SpringBoot`如何集成`kafka`，實現消息的發送和接收

版本使用的是Boot是:2.0.6 kafka版本是: 2.1.10

環境準備,使用容器(`docker`)安裝部署`kafka`

1. 下載鏡像

kafka需要zookeeper管理，所以需要先安裝zookeeper鏡像。 docker pull wurstmeister/zookeeper
然後安裝kafka鏡像: docker pull wurstmeister/kafka

2. 啓動`zookeeper`和`kafka`容器

啓動zookeeper鏡像docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper
啓動kafka鏡像生成容器

docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=10.0.75.1:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.0.75.1:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka

-e KAFKA_BROKER_ID=0 在kafka集羣中，每個kafka都有一個BROKER_ID來區分自己
-e KAFKA_ZOOKEEPER_CONNECT=10.0.75.1:2181/kafka 配置zookeeper管理kafka的路徑192.168.155.56:2181/kafka (這裏同一個虛擬機ip)
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.0.75.1:9092 把kafka的地址端口註冊給zookeeper (這裏同一個虛擬機ip)
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 配置kafka的監聽端口

3. 進行測試是否部署成功

進入kafka容器的命令行: docker exec -it kafka /bin/bash
進入kafka啓動命令所在bin目錄: cd opt/kafka_x.xx-x.x.x/bin

運行kafka生產者發送消息: ./kafka-console-producer.sh --broker-list localhost:9092 --topic mykafka

{"datas":[{"channel":"","metric":"temperature","producer":"ijinus","sn":"IJA0101-00002245","time":"1543207156000","value":"80"}],"ver":"1.0"}

4.運行kafka消費者接收消息 : kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mykafka --from-beginning

到此容器部署併成功啓動了kafka

SpringBoot集成kafka

創建Kafka主題 - Kafka提供了一個名爲 kafka-topics.sh 的命令行實用程序，用於在服務器上創建主題。打開新終端並創建一個主題一個名爲test的Topic

先進到opt/kafka_x.xx-x.x.x/bin目錄

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

這條命令的意思是，創建一個Topic到ZK（指定ZK的地址），副本個數爲1，分區數爲1，Topic的名稱爲test。

1. 導入依賴:

 	<parent>
        <artifactId>spring-boot-demo-base</artifactId>
        <groupId>spring-boot-demo-base</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>spring-boot-demo-kafka</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <dependencies>
        <!--spring-kafka 依賴-->
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>
        <!--lombok-->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>

2. `application.yml`

server:
  port: 8080
  servlet:
    context-path: /kafka
spring:
  kafka:
    bootstrap-servers: 10.0.75.1:9092
    #如果只需要發送信息就只配置生產者
    producer:
      retries: 0
      batch-size: 16384
      buffer-memory: 33554432
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    #消費者配置
    consumer:
      group-id: test-consumer
      # 手動提交
      enable-auto-commit: false
      auto-offset-reset: latest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      properties:
        session.timeout.ms: 60000
    listener:
      log-container-config: false
      concurrency: 5
      # 手動提交
      ack-mode: manual_immediate

3. `KafkaConfig.java`配置類

/**
 * @Author: zhihao
 * @Date: 8/1/2020 下午 8:31
 * @Description: kafka配置類
 * @Versions 1.0
 **/
@Configuration
@EnableConfigurationProperties({KafkaProperties.class})
@EnableKafka
@AllArgsConstructor
public class KafkaConfig {

    private final KafkaProperties kafkaProperties;

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        return new DefaultKafkaProducerFactory<>(kafkaProperties.buildProducerProperties());
    }

    //------------------------------以下是消費者配置-----------------

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(3);
        factory.setBatchListener(true);
        factory.getContainerProperties().setPollTimeout(3000);
        return factory;
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(kafkaProperties.buildConsumerProperties());
    }

    @Bean("ackContainerFactory")
    public ConcurrentKafkaListenerContainerFactory<String, String> ackContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL_IMMEDIATE);
        factory.setConcurrency(3);
        return factory;
    }
}

4. 寫個消費者`KafkaMessageHandler`監聽消息

/**
 * @Author: zhihao
 * @Date: 8/1/2020 下午 9:19
 * @Description: 消息消費者
 * @Versions 1.0
 **/
@Component
@Slf4j
public class KafkaMessageHandler {

    /***
     * 接收消息後手動提交
     *
     * @param record 消費記錄
     * @param acknowledgment 確認接收
     * @return void
     * @author: zhihao
     * @date: 8/1/2020
     */

    @KafkaListener(topics = "test",containerFactory = "kafkaListenerContainerFactory")
    public void handlerMessage(ConsumerRecord record, Acknowledgment acknowledgment){
        try {
            //手動接收消息
            String value = (String) record.value();
            System.out.println("手動接收<<接收到消息,進行消費>>>"+value);
        } catch (Exception e) {
            log.error("手動接收<<消費異常信息>>>"+e.getMessage());
        }finally {
            //最終提交確認接收到消息  手動提交 offset
            acknowledgment.acknowledge();
        }
    }

//    /***
//     * 接收消息後自動提交 需要配置開啓enable-auto-commit: true
//     *
//     * @param message 消息
//     * @return void
//     * @author: zhihao
//     * @date: 8/1/2020
//     */
//    @KafkaListener(topics = "test",groupId = "test-consumer")
//    public void handlerMessage(String message){
//        System.out.println("接收到自動確認消息"+message);
//    }
}

5. 寫測試類`KafkaSendMessage`進行發送消息:

/**
 * @Author: zhihao
 * @Date: 8/1/2020 下午 9:36
 * @Description: 測試發送消息
 * @Versions 1.0
 **/
@SpringBootTest(classes = ApplicationKafka.class)
@RunWith(value = SpringRunner.class)
public class KafkaSendMessage {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    /***
     * 簡單發送消息
     *
     * @param message 消息
     * @return
     * @author: zhihao
     * @date: 8/1/2020
     */
    public void testSend(String message){
        //向test主題發送消息
        kafkaTemplate.send("test",message);
    }


    /***
     * 發送消息獲取發送成功或者失敗
     *
     * @param message 消息
     * @return
     * @author: zhihao
     * @date: 8/1/2020
     */
    public void Send(String message){
        //向test主題發送消息
        ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send("test", message);
        future.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
            @Override
            public void onFailure(Throwable throwable) {
                System.out.printf("消息：{} 發送失敗，原因：{}", message, throwable.getMessage());
            }

            @Override
            public void onSuccess(SendResult<String, String> stringStringSendResult) {
                System.out.printf("成功發送消息：{}，offset=[{}]", message, stringStringSendResult.getRecordMetadata().offset());
            }
        });

    }

    @Test
    public void test(){
        this.testSend("這是一個簡單發送消息測試");

        this.Send("這是一個發送消息獲取發送結果測試");
    }
}

擴展資料:

編程獅文檔:

Git-Kafka自動提交消息教程

Spring Boot 版本和 Spring-Kafka 的版本對應關係：https://spring.io/projects/spring-kafka

Spring-Kafka 官方文檔：https://docs.spring.io/spring-kafka/docs/2.2.0.RELEASE/reference/html/

Kafka 官方文檔 : http://kafka.apache.org/

項目代碼:

SpringBoot安裝集成kafka,實現消息的發送和接收

`SpringBoot`如何集成`kafka`，實現消息的發送和接收

環境準備,使用容器(`docker`)安裝部署`kafka`

1. 下載鏡像

2. 啓動`zookeeper`和`kafka`容器

3. 進行測試是否部署成功

SpringBoot集成kafka

1. 導入依賴:

2. `application.yml`

3. `KafkaConfig.java`配置類

4. 寫個消費者`KafkaMessageHandler`監聽消息

5. 寫測試類`KafkaSendMessage`進行發送消息:

擴展資料:

DAPPER 事務 TRANSACTION

Hystrix監控面板與監控數據聚合(學習筆記2020.03.11)

SpringBoot中使用註解方式進行多線程異步(學習筆記2020.3.30)

三大框整合複習筆記(Spring+SpringMVC+Mybatis)

Kafka消息隊列學習筆記(2020.3.20)

SrpingBoot-Shrio 整合 JWT (7)-不夠完善

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

SpringBoot安裝集成kafka,實現消息的發送和接收

SpringBoot如何集成kafka，實現消息的發送和接收

環境準備,使用容器(docker)安裝部署kafka

1. 下載鏡像

2. 啓動zookeeper和kafka容器

3. 進行測試是否部署成功

SpringBoot集成kafka

1. 導入依賴:

2. application.yml

3. KafkaConfig.java配置類

4. 寫個消費者KafkaMessageHandler監聽消息

5. 寫測試類KafkaSendMessage進行發送消息:

擴展資料:

`SpringBoot`如何集成`kafka`，實現消息的發送和接收

環境準備,使用容器(`docker`)安裝部署`kafka`

2. 啓動`zookeeper`和`kafka`容器

2. `application.yml`

3. `KafkaConfig.java`配置類

4. 寫個消費者`KafkaMessageHandler`監聽消息

5. 寫測試類`KafkaSendMessage`進行發送消息: