python使用kafka收發消息

簡介

kafka是最近幾年很流行的消息隊列中間件。在大數據以及後端服務領域有很廣泛的應用。廢話不多說，接下來直接上代碼介紹python如何向kafka發送數據以及訂閱數據。

消息發佈

Kafka的消息是 " 發佈--訂閱" 模式的。 接下來先介紹向kakfa發佈消息。先安裝python的kafka連接模塊。pip install kafka-python.

 

import time
from kafka import KafkaProducer


class KafkaMsgProducer:
    def __init__(self, server):
        self._server = server
        self.producer = None

    def connect(self):
        if self.producer is None:
            producer = KafkaProducer(bootstrap_servers=self._server)
            self.producer = producer

    def close(self):
        if self.producer is not None:
            self.producer.close()
            self.producer = None

    def send(self, topic, msg):
        if self.producer is not None:
            if not isinstance(msg, bytes):
                msg = msg.encode("utf-8")  # 將str類型轉換爲bytes類型
            self.producer.send(topic=topic, value=msg)


def run():
    producer = KafkaMsgProducer("localhost:9092")
    producer.connect()  # 建立連接
    topic = "YanChampion-Test"

    print("Start sending msg to kafka!")

    for msg in "Hello! This is YanChampion speaking!".split():
        producer.send(topic=topic, msg=msg)   # 向kafka 指定topic發送數據
        time.sleep(1)


if __name__ == '__main__':
    run()    # 運行發佈消息程序

以上代碼即可以向kafka指定topic發佈消息了。注意，爲了測試，先不運行producer.py

消息訂閱

 

 import time
from kafka import KafkaProducer


class KafkaMsgProducer:
    def __init__(self, server):
        self._server = server
        self.producer = None

    def connect(self):
        if self.producer is None:
            producer = KafkaProducer(bootstrap_servers=self._server)
            self.producer = producer

    def close(self):
        if self.producer is not None:
            self.producer.close()
            self.producer = None

    def send(self, topic, msg):
        if self.producer is not None:
            if not isinstance(msg, bytes):
                msg = msg.encode("utf-8")  # 將str類型轉換爲bytes類型
            self.producer.send(topic=topic, value=msg)


def run():
    producer = KafkaMsgProducer("localhost:9092")
    producer.connect()  # 建立連接
    topic = "YanChampion-Test"

    print("Start sending msg to kafka!")

    for msg in " 111 222 333 444".split():
        producer.send(topic=topic, msg=msg)   # 向kafka 指定topic發送數據
        time.sleep(1)


if __name__ == '__main__':
    run()    # 運行程序

以上代碼即可完成消息的訂閱。

測試

因爲kafka是 發佈-定於模式。所以，一個topic可以有多個consumer訂閱，並且，每個consumer都可以收到同一條消息。那麼讓我們先來運行兩個consumer.py文件。
打開不同的終端
python3 consumer.py
接下來再運行producer.py
python3 producer.py

通過觀察，最終可以看到，兩個運行了consumer.py 的終端 都可以收到消息

 

[root@yanchampion kafka-demo]# python3 consumer.py 
收到消息： b'111'
收到消息： b'222'
收到消息： b'333'
收到消息： b'444'