librdkafka 打印連接斷開錯誤

1. librdkafka日誌

2016-01-24 08:56:58.773506|0x000022a4|error|gg::lqs::kafka::KafkaProducer::event_cb|KafkaProducer.cpp:363 FAIL: localhost:9092/bootstrap: Connection closed
2016-01-24 08:56:58.787507|0x00002a60|error|gg::lqs::kafka::KafkaProducer::event_cb|KafkaProducer.cpp:363 Error event: -195: Local: Broker transport failure - localhost:9092/bootstrap: Connection closed

2. kafka 日誌

less ../log/kafka.log | grep "Removed 0 expired offsets in 0 milliseconds."
[2016-07-08 05:11:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:21:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:31:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:41:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 05:51:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:01:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:11:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:21:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:31:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:41:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 06:51:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:01:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:11:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:21:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:31:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-07-08 07:41:35,940] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)

3. 現象說明

經過測試，kafka中並不會少數據。

kafka 0.9.0+ 版本會每隔一段時間主動關閉空閒連接，默認是10分鐘。上面kafka也是每隔十分鐘纔會出現特定的日誌信息。

在librdkafka中，有一個決定是否打印斷連信息的開關。如果對接的是0.9.0+版本的kafka，建議關閉。在CONFIGURATION.md中有如下一段說明，該開關默認是開啓的。

log.connection.close                     |  *  |          true | Log broker disconnects. It might be useful to turn this off when interacting with 0.9 brokers with an aggressive `connection.max.idle.ms` value.

關閉該開關的辦法：在調用librdkafka創建kafka實例時，增加配置。

if (rd_kafka_conf_set(rk_conf, "log.connection.close", "false", errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {
    writeLog("set log.connection.close failed.\n");
}

rk = rd_kafka_new(RD_KAFKA_PRODUCER, rk_conf, errstr, sizeof(errstr));

4. github issue

https://github.com/edenhill/librdkafka/issues/516