RocketMQ源碼分析——消息的拉取

創建消息消費者

實例化消費者、設置NameServer的地址、設置消費起始位置（獲取消費進度失敗時有效）、訂閱一個或者多個 Topic 以及 Tag 來過濾需要消費的消息、註冊回調實現類來處理從broker拉取回來的消息、啓動消費者。
org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#start

...
// 創建訂閱消息 SubscriptionData 到負載服務中
this.copySubscription();
// 廣播模式消息消費進度由消費者控制，集羣模式由Broker控制
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case BROADCASTING:
            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        case CLUSTERING:
            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        default:
            break;
    }
    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
// 加載消息進度
this.offsetStore.load();
// 區分是併發消費還是順序消費
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
    this.consumeOrderly = true;
    this.consumeMessageService =
        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
    this.consumeOrderly = false;
    this.consumeMessageService =
        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
...
// 更新所有主題訂閱信息
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
// 校驗所有的訂閱配置是否正確
this.mQClientFactory.checkClientInBroker();
// 向所有Broker發送心跳
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
// 立即進行一次消費者負載
this.mQClientFactory.rebalanceImmediately();

消息拉取請求

客戶端實例 MQClientInstance 中有一個單獨的線程 PullMessageService 來定時拉取消息。

public void run() {
    while (!this.isStopped()) {
        try {
        	// 從 LinkedBlockingQueue 中獲取一個拉取請求，沒有就阻塞
        	// 拉取請求在客戶端做負載均衡時會創建，一個拉取任務完成後還會繼續拉取。
            PullRequest pullRequest = this.pullRequestQueue.take();
            this.pullMessage(pullRequest);
        } catch (InterruptedException ignored) {
        } catch (Exception e) {
        }
    }
}

調用拉取消息方法，先校驗消費隊列狀態，進行流控
org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage

public void pullMessage(final PullRequest pullRequest) {
	final ProcessQueue processQueue = pullRequest.getProcessQueue();
	// 隊列被刪除，禁止消費
	if (processQueue.isDropped()) {
	    return;
	}
	pullRequest.getProcessQueue().setLastPullTimestamp(System.currentTimeMillis());
	try {
		// 客戶端服務服務是否正常
	    this.makeSureStateOK();
	} catch (MQClientException e) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
	    return;
	}
	// 隊列流控，默認延遲1s之後再拉取消息
	if (this.isPause()) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_SUSPEND);
	    return;
	}
	// 待消費的消息總數量
	long cachedMessageCount = processQueue.getMsgCount().get();
	// 待消費的消息總大小
	long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024);
	// 默認超過1000條進行流控
	if (cachedMessageCount > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
	    return;
	}
	// 默認超過100M進行流控
	if (cachedMessageSizeInMiB > this.defaultMQPushConsumer.getPullThresholdSizeForQueue()) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
	    return;
	}
    if (!this.consumeOrderly) {
    	// 非順序消費，默認消息的偏移量間隔超過2000，進行流控
        if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
            this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
            return;
        }
    } else {
    	// 順序消費，需要先判斷是否有鎖定
        if (processQueue.isLocked()) {
            if (!pullRequest.isLockedFirst()) {
            	// 第一次鎖定記錄下次拉取的偏移量
                final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
                boolean brokerBusy = offset < pullRequest.getNextOffset();

                pullRequest.setLockedFirst(true);
                pullRequest.setNextOffset(offset);
            }
        } else {
        	// 默認延遲3s再拉取
            this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
            return;
        }
    }
}

構造請求參數向Broker發送拉取請求
org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#pullKernelImpl

public PullResult pullKernelImpl(
    final MessageQueue mq, //消費隊列
    final String subExpression, //過濾表達式
    final String expressionType, //表達式類型 TAG/SQL
    final long subVersion, //過濾信息版本
    final long offset, //拉取偏移量位置
    final int maxNums, //最大拉取數量
    final int sysFlag, //系統標識
    final long commitOffset, //當前消費進度
    final long brokerSuspendMaxTimeMillis, //默認掛起時間15s
    final long timeoutMillis, //默認超時時間30s
    final CommunicationMode communicationMode, //默認異步
    final PullCallback pullCallback //回調函數
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
	...
	String brokerAddr = findBrokerResult.getBrokerAddr();
	// 如果是類過濾，獲取到類過濾服務器的地址
	if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
	    brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
	}
	...
}

服務端查找消息

org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest

private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend)
        throws RemotingCommandException {
	...
	// 校驗Broker是否允許拉取消息、過濾主題信息是否存在和允許拉取、主題是否存在和允許拉取、隊列ID是否正確、過濾表達式或者是否正確是否存在
	// 根據偏移量查找消息結果，消息過濾的邏輯之後分析
	final GetMessageResult getMessageResult =
            this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
                requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);
	// 根據結果，返回各種狀態碼，此處忽略
	switch (response.getCode()) {
	   case ResponseCode.SUCCESS:
			if (this.brokerController.getBrokerConfig().isTransferMsgByHeap()) {
				// 通過內存處理所有的消息
                final byte[] r = this.readGetMessageResult(getMessageResult, requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId());
                response.setBody(r);
            } else {
            	...
            	//使用netty的異步通信
            }
		case ResponseCode.PULL_NOT_FOUND:
			// 沒有找到消息
		    if (brokerAllowSuspend && hasSuspendFlag) {
		    	// Push模式實現的關鍵點就是這裏
		        long pollingTimeMills = suspendTimeoutMillisLong;
		        if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
		            pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
		        }
		
		        String topic = requestHeader.getTopic();
		        long offset = requestHeader.getQueueOffset();
		        int queueId = requestHeader.getQueueId();
		        // 創建 PullRequest 對象
		        PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
		            this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter);
		        // 交給 PullRequestHoldService 異步處理拉取請求
		        this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
		        // 返回null，不向 channel 寫入數據，相當於掛起當前請求
		        response = null;
		        break;
		    }
}

RocketMQ 的 Consumer 都是從 Broker 拉消息來消費，但是爲了能做到實時收消息，RocketMQ 使用長輪詢方式，可以保證消息實時性同 Push 方式一致。Broker 單獨開啓一個線程來 hold 客戶端拉取請求。
org.apache.rocketmq.broker.longpolling.PullRequestHoldService#run

public void run() {
    while (!this.isStopped()) {
        try {
            if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
            	// 開啓長輪詢，每5秒嘗試一次
                this.waitForRunning(5 * 1000);
            } else {
            	// 未開啓長輪詢，默認1s嘗試一次
                this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
            }
            // 有滿足條件的消息就返回
            this.checkHoldRequest();
        } catch (Throwable e) {
        }
    }
}

checkHoldRequest 會遍歷所有的拉取請求，執行 notifyMessageArriving
org.apache.rocketmq.broker.longpolling.PullRequestHoldService#notifyMessageArriving

	...
	// 獲取 CommitLog 中最大的物理偏移量 newestOffset，比本次拉取值大代表有新消息存儲進來了
	if (newestOffset > request.getPullFromThisOffset()) {
		// 過濾校驗消息通過後，喚醒被掛起的拉取線程
		brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
        	request.getRequestCommand());
	}
	if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
		// 超時了，也需要喚醒
		...
	}
	...

長輪詢機制：每5s檢查一次是否有滿足過濾條件的新消息，超過15s仍然沒有就返回超時。Broker 在有新消息進入 CommitLog 後，分發任務 ReputMessageService 也會執行 notifyMessageArriving 喚醒拉取任務。
org.apache.rocketmq.store.DefaultMessageStore.ReputMessageService#doReput

	...
	if (BrokerRole.SLAVE != DefaultMessageStore.this.getMessageStoreConfig().getBrokerRole()
	    && DefaultMessageStore.this.brokerConfig.isLongPollingEnable()) {
	    DefaultMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
	        dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
	        dispatchRequest.getTagsCode(), dispatchRequest.getStoreTimestamp(),
	        dispatchRequest.getBitMap(), dispatchRequest.getPropertiesMap());
	}

消息隊列負載

負載原則：一個消息消費隊列在同一時間只允許被同一消費組內的一個消費者消費，一個消息消費者能同時消費多個消息隊列。
MQClientInstance 中有一個單獨的 RebalanceService 線程來定時做負載

public void run() {
    while (!this.isStopped()) {
    	// 默認間隔20s
        this.waitForRunning(waitInterval);
        this.mqClientFactory.doRebalance();
    }
}

public void doRebalance(final boolean isOrder) {
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
    if (subTable != null) {
    	// 遍歷每個主題，做負載
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                this.rebalanceByTopic(topic, isOrder);
            } catch (Throwable e) {
            }
        }
    }
	// 清除沒有訂閱的主題下的消息隊列
    this.truncateMessageQueueNotMyTopic();
}

集羣模式下，消息由所有消費者共同消費，怎麼確定消費隊列由那個消費者消費？
org.apache.rocketmq.client.impl.consumer.RebalanceImpl#rebalanceByTopic

private void rebalanceByTopic(final String topic, final boolean isOrder) {
	...
	// 從Broker獲取主題下有哪些消費者
	List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
	// 排序，保證所有消費者客戶端看到的數據一致
    Collections.sort(mqAll);
    Collections.sort(cidAll);
    AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
    List<MessageQueue> allocateResult = null;
    try {
    	// 執行負載算法，計算當前消費者應該消費的隊列
        allocateResult = strategy.allocate(
            this.consumerGroup,
            this.mQClientFactory.getClientId(),
            mqAll,
            cidAll);
    } catch (Throwable e) {
        return;
    }
}

負載算法，默認爲平均分配 AllocateMessageQueueAveragely

清除舊隊列，新增新加入的隊列請求

    private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
        final boolean isOrder) {
        boolean changed = false;
		// 清除沒有分配給自己的隊列
        Iterator<Entry<MessageQueue, ProcessQueue>> it = this.processQueueTable.entrySet().iterator();
        while (it.hasNext()) {
            Entry<MessageQueue, ProcessQueue> next = it.next();
            MessageQueue mq = next.getKey();
            ProcessQueue pq = next.getValue();

            if (mq.getTopic().equals(topic)) {
                if (!mqSet.contains(mq)) {
                    pq.setDropped(true);
                    if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                        it.remove();
                        changed = true;
                        log.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
                    }
                } else if (pq.isPullExpired()) {
                    // 默認120s過期，push模式纔有效，清除消費隊列數據
                }
            }
        }
		// 新分配隊列給自己，新建 ProcessQueue 
        List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
        for (MessageQueue mq : mqSet) {
            if (!this.processQueueTable.containsKey(mq)) {
            	// 順序消息需要判斷是否有鎖
                if (isOrder && !this.lock(mq)) {
                    continue;
                }
				// 移除緩存中此隊列的消費進度
                this.removeDirtyOffset(mq);
                ProcessQueue pq = new ProcessQueue();
                // 新隊列需要計算從哪個位置開始拉取
                long nextOffset = this.computePullFromWhere(mq);
                if (nextOffset >= 0) {
                    ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                    if (pre != null) {
                    } else {
                        PullRequest pullRequest = new PullRequest();
                        pullRequest.setConsumerGroup(consumerGroup);
                        pullRequest.setNextOffset(nextOffset);
                        pullRequest.setMessageQueue(mq);
                        pullRequest.setProcessQueue(pq);
                        pullRequestList.add(pullRequest);
                        changed = true;
                    }
                } else {
                }
            }
        }
		// 將請求添加到 pullRequestQueue
        this.dispatchPullRequest(pullRequestList);
        return changed;
    }

consumeFromWhere 參數的作用

新增加的消費隊列，移除緩存中此隊列的消費進度後，需要從磁盤中加載此隊列的消費進度，然後創建 PullRequest 請求。
consumeFromWhere 只在從磁盤獲取消費進度失敗才生效。
偏移量計算邏輯，以PUSH模式爲例，消費進度存儲在Broker。

    public long computePullFromWhere(MessageQueue mq) {
        long result = -1;
        final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
        final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
        switch (consumeFromWhere) {
            case CONSUME_FROM_LAST_OFFSET: {
            	// 讀取Broker存儲的進度
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                }
                // First start,no offset
                else if (-1 == lastOffset) {
                    if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        result = 0L;
                    } else {
                        try {
                        	// 讀取隊列的最大偏移量
                            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    }
                } else {
                    result = -1;
                }
                break;
            }
            case CONSUME_FROM_FIRST_OFFSET: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                } else if (-1 == lastOffset) {
                    result = 0L;
                } else {
                    result = -1;
                }
                break;
            }
            case CONSUME_FROM_TIMESTAMP: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                } else if (-1 == lastOffset) {
                    if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        try {
                            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    } else {
                        try {
                        	// 以消費者的啓動時間來獲取消費隊列的偏移量
                            long timestamp = UtilAll.parseDate(this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeTimestamp(),
                                UtilAll.YYYYMMDDHHMMSS).getTime();
                            // 從 commitLog 讀取消息的存儲時間，取存儲時間和 timestamp 最接近的消息的偏移量，使用了二分法
                            result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    }
                } else {
                    result = -1;
                }
                break;
            }

            default:
                break;
        }

        return result;
    }

消息消費

拉取消息獲取到結果後，按照服務端返回的狀態封裝拉取結果 PullResult
org.apache.rocketmq.client.impl.MQClientAPIImpl#processPullResponse

        switch (response.getCode()) {
            case ResponseCode.SUCCESS:
                pullStatus = PullStatus.FOUND;
                break;
            case ResponseCode.PULL_NOT_FOUND:
                pullStatus = PullStatus.NO_NEW_MSG;
                break;
            case ResponseCode.PULL_RETRY_IMMEDIATELY:
                pullStatus = PullStatus.NO_MATCHED_MSG;
                break;
            case ResponseCode.PULL_OFFSET_MOVED:
                pullStatus = PullStatus.OFFSET_ILLEGAL;
                break;

            default:
                throw new MQBrokerException(response.getCode(), response.getRemark());
        }

再回到 org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage
執行拉取任務的回調函數 PullCallback

        PullCallback pullCallback = new PullCallback() {
            @Override
            public void onSuccess(PullResult pullResult) {
                if (pullResult != null) {
                	// tag過濾，還需要再校驗一次原始的tag值，Broker只校驗了hashcode，不讀消息內容保證堆積也能高效過濾
                    pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                        subscriptionData);

                    switch (pullResult.getPullStatus()) {
                        case FOUND:
                            // 記錄拉取統計信息
                            	// 將消息數據放入到 processQueue
                                boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                                // 提交一個消息消費任務，區分並行還是順序
                                DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                                    pullResult.getMsgFoundList(),
                                    processQueue,
                                    pullRequest.getMessageQueue(),
                                    dispatchToConsume);

                                // 設置偏移量，再次拉取
                            }
                            break;
                        case NO_NEW_MSG:
                        	// 修正偏移量，再次拉取
                        case NO_MATCHED_MSG:
                        	// 修正偏移量，再次拉取
                        case OFFSET_ILLEGAL:
                            // 刪除此消費隊列，更新並持久化消費進度
                            break;
                        default:
                            break;
                    }
                ...

消息拉取總結：啓動消費者，消費者按照負載均衡策略生成拉取消息請求，設置拉取模式，讀取消費進度，設置拉取起始偏移量，然後定時向Broker拉取消息。Broker接受拉取請求，從CommitLog查找消息，按照過濾策略處理消息後返回給消費者。若拉取任務失敗，消息消費者修正偏移量再次拉取；若拉取成功，消費者進行消費，並更新消費進度，再次拉取。

消息消費詳情、消息確認、消息進度管理見後續文章。