docker搭建kafka

  分佈式爬蟲框架的部分功能已經開發的差不多了,現在要開始考慮接入kafka了.

  kafka依賴於zookeeper, 所以同時需要安裝zookeeper

   

  dockerfile-kafka如下

  

FROM java:8u111-jdk

MAINTAINER stcoder

RUN apt-get update

RUN apt-get -y install wget tar supervisor

WORKDIR /tmp

RUN wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz

RUN mkdir /home/kafka

RUN tar zxvf kafka_2.11-1.0.0.tgz -C /home

ADD supervisord.conf /etc/supervisord.conf

EXPOSE 2181 2888 3888 9092

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf"]

這裏要說個東西了-----supervisor,我在這個容器裏啓動了兩個服務,zookeeper與kafka,這就是依靠supervisor實現的,下面是supervisor.conf

[supervisord]
nodaemon=true

[program:zookeeper]
command=/home/kafka_2.11-1.0.0/bin/zookeeper-server-start.sh /home/kafka_2.11-1.0.0/config/zookeeper.properties

[program:kafka]
command=/home/kafka_2.11-1.0.0/bin/kafka-server-start.sh /home/kafka_2.11-1.0.0/config/server.properties

接下來是docker-compose.yml

version: '2'
services:
  kafka:
    restart: always
    build:
      context: .
      dockerfile: Dockerfile-kafka
    volumes:
      # 用於將自定義的配置掛載進入容器
      - ./conf/server.properties:/home/kafka_2.11-1.0.0/config/server.properties:ro
      - ./conf/zookeeper.properties:/home/kafka_2.11-1.0.0/config/zookeeper.properties:ro

這裏將同級目錄下的conf文件夾裏的配置文件掛載到容器的卷,從而實現自定義配置

到這裏爲止,一個單機版本的kafka容器已經構建完成了,接下來就是進行測試了,我們用flask構建一個web服務,用戶請求/test時向topic名爲mytopic的topic發送一個負載爲hello的消息,test-app.py如下

from flask import Flask, request
from confluent_kafka import Producer

app = Flask(__name__)
kafka_conf = {'bootstrap.servers': 'kafka:9092'}

@app.route('/test')
def test():
    p = Producer(kafka_conf)
    p.produce('mytopic', 'hello')
    p.flush()
    return 'ok'

if __name__ == '__main__':
    app.run('0.0.0.0')

然後構建這個webserver的dockerfile-app如下

FROM ubuntu:16.04

MAINTAINER stcoder

RUN apt-get update

RUN apt-get -y install python3 python3-dev python3-pip \
wget software-properties-common python-software-properties

RUN wget -qO - http://packages.confluent.io/deb/4.0/archive.key | apt-key add -
RUN add-apt-repository "deb [arch=amd64] http://packages.confluent.io/deb/4.0 stable main"
RUN apt-get update && apt-get -y install librdkafka-dev

ADD requirements.txt /tmp/requirements.txt

WORKDIR /tmp

RUN pip3 install --upgrade pip

RUN pip3 install -r requirements.txt

WORKDIR /home

RUN mkdir kafka-docker-test

WORKDIR kafka-docker-test

CMD python3 test-app.py

然後整個測試系統的構建的docker-compose.yml如下

version: '2'
services:
  kafka:
    restart: always
    build:
      context: .
      dockerfile: Dockerfile-kafka
    volumes:
      # 用於將自定義的配置掛載進入容器
      - ./conf/server.properties:/home/kafka_2.11-1.0.0/config/server.properties:ro
      - ./conf/zookeeper.properties:/home/kafka_2.11-1.0.0/config/zookeeper.properties:ro
  app:
    restart: always
    build:
      context: .
      dockerfile: Dockerfile-app
    volumes:
      - ./:/home/kafka-docker-test:ro
    ports:
      - "5000:5000"
    links:
      - kafka

整個測試項目的目錄結構如下

.
├── conf
│   ├── server.properties
│   └── zookeeper.properties
├── docker-compose.yml
├── Dockerfile-app
├── Dockerfile-kafka
├── requirements.txt
├── supervisord.conf
└── test-app.py

啓動docker-compose.yml up後訪問127.0.0.1:5000/test
頁面返回ok

使用docker exec -it <kafka容器> /bin/bash

進入kafka根目錄, 

執行bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
mytopic

顯示mytopic創建成功了,測試到此爲止,kafka容器可用