46. 源代碼解讀-RocketMQ-namesrv

一. Namesrv作用

Namesrv，命名服務，顧名思義，應該是做服務管理之類的工作。比如管理所有的mq broker地址，狀態那些。

事實上，也確實如此。

Namesrv做了如下工作：

維護所有mq broker的狀態，定期掃描broker的狀態。如果在一定時間內(2分鐘)沒有發起註冊動作(REGISTER_BROKER)，那麼將清除這個broker裏面的數據。
Broker註冊和取消註冊。
這是namesrv接收到的一次Broker請求消息，103代表請求的類型是註冊Broker

public static final int REGISTER_BROKER = 103;

2017-12-01 17:52:49 INFO RemotingExecutorThread_3 - request code: 103
2017-12-01 17:52:49 INFO RemotingExecutorThread_3 - request.getBody(): {"filterServerList":[],"topicConfigSerializeWrapper":{"dataVersion":{"counter":5,"timestamp":1511955797449},"topicConfigTable":{"huangrongweideMacBook-Pro.local":{"order":false,"perm":7,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"huangrongweideMacBook-Pro.local","topicSysFlag":0,"writeQueueNums":1},"SELF_TEST_TOPIC":{"order":false,"perm":6,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"SELF_TEST_TOPIC","topicSysFlag":0,"writeQueueNums":1},"%RETRY%lalaase_rename_unique_group_name_4":{"order":false,"perm":6,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"%RETRY%lalaase_rename_unique_group_name_4","topicSysFlag":0,"writeQueueNums":1},"DefaultCluster":{"order":false,"perm":7,"readQueueNums":16,"topicFilterType":"SINGLE_TAG","topicName":"DefaultCluster","topicSysFlag":0,"writeQueueNums":16},"mmmzzz":{"order":false,"perm":6,"readQueueNums":4,"topicFilterType":"SINGLE_TAG","topicName":"mmmzzz","topicSysFlag":0,"writeQueueNums":4},"wangyuan.freecomm-networks.com":{"order":false,"perm":7,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"wangyuan.freecomm-networks.com","topicSysFlag":0,"writeQueueNums":1},"%RETRY%bybybse_rename_unique_group_name_4":{"order":false,"perm":6,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"%RETRY%bybybse_rename_unique_group_name_4","topicSysFlag":0,"writeQueueNums":1},"%RETRY%phihomebse_rename_unique_group_name_4":{"order":false,"perm":6,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"%RETRY%phihomebse_rename_unique_group_name_4","topicSysFlag":0,"writeQueueNums":1},"TBW102":{"order":false,"perm":7,"readQueueNums":8,"topicFilterType":"SINGLE_TAG","topicName":"TBW102","topicSysFlag":0,"writeQueueNums":8},"BenchmarkTest":{"order":false,"perm":6,"readQueueNums":1024,"topicFilterType":"SINGLE_TAG","topicName":"BenchmarkTest","topicSysFlag":0,"writeQueueNums":1024},"OFFSET_MOVED_EVENT":{"order":false,"perm":6,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"OFFSET_MOVED_EVENT","topicSysFlag":0,"writeQueueNums":1},"%RETRY%please_rename_unique_group_name_4":{"order":false,"perm":6,"readQueueNums":1,"topicFilterType":"SINGLE_TAG","topicName":"%RETRY%please_rename_unique_group_name_4","topicSysFlag":0,"writeQueueNums":1}}}}

維護topic和它對應的broker映射關係，由於rocketmq裏面各個master broker是獨立的，所以必然有個東西來維護topic和broker的關係。
broker會定時上報broker的信息，包括它包含的topic。然後namesrv就會比較記錄這些topic和broker的對應關係。
查詢Broker以及集羣信息
提供topic路由信息查詢topic對應的broker信息。
key-value操作，包括key-value的保存，查詢以及刪除等操作。
那每次Broker都攜帶信息過來，namesrv每次都會更新效率不是更低？

Broker定期會向namesrv註冊，同時會攜帶它所有的topic信息以及數據版本dataVersion，每次有topic更新，從Broker傳過來的dataVersion都會+1
namesrv 根據dataVersion判斷是否需要更新namesrv。
所以使用Rocketmq的時候,topic不要太多。幾十幾百還是可以的，但是成千上萬的topic的話，別的不說，光從broker通過網絡發送過來，都是不小的開支。

二. 流程圖

三. 源代碼流程分析

1. mqname.sh

啓動namesrv的入口是執行sh bin/mqname &，所以先看mqname.sh內容

...

export ROCKETMQ_HOME

sh ${ROCKETMQ_HOME}/bin/runserver.sh org.apache.rocketmq.namesrv.NamesrvStartup $@

前面一系列的初始化之後，然後調用NamesrvStartup.java的main函數

2. NamesrvStartup

public static void main(String[] args) {
    main0(args);
}

public static NamesrvController main0(String[] args) {
       ...

       final NamesrvController controller = new NamesrvController(namesrvConfig, nettyServerConfig);

       ...

       boolean initResult = controller.initialize();
       ...

       controller.start();
       ...
   } catch (Throwable e) {

   }

  return null;
}

3. NamesrvController.initialize

從startup->Controller，在RocketMQ裏面很多都是這種代碼組織方式(同一個人寫的？)

private void registerProcessor() {
    ...
        this.remotingServer.registerDefaultProcessor(new DefaultRequestProcessor(this), this.remotingExecutor);
}

public boolean initialize() {

        this.kvConfigManager.load();

        this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);

        ...

        this.registerProcessor();

        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                NamesrvController.this.routeInfoManager.scanNotActiveBroker();
            }
        }, 5, 10, TimeUnit.SECONDS);

       ...

       return true;
}

initialize方法首先初始化NettyRemotingServer變量，NettyRemotingServer是RocketMQ-remoting裏面的類，作用是接受Netty網絡調用請求，然後回調相應的接口，這裏是回調DefaultRequestProcessor

啓動一個定時執行的定時器10秒

NamesrvController.this.routeInfoManager.scanNotActiveBroker();

接下來，我們分析下scanNotActiveBroker是幹什麼的。

4. RouteInfoManager.scanNotActiveBroker

private final static long BROKER_CHANNEL_EXPIRED_TIME = 1000 * 60 * 2;

public void scanNotActiveBroker() {
        Iterator<Entry<String, BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
        while (it.hasNext()) {
            Entry<String, BrokerLiveInfo> next = it.next();
            long last = next.getValue().getLastUpdateTimestamp();
            if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
                RemotingUtil.closeChannel(next.getValue().getChannel());
                it.remove();
                log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
                this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
            }
        }
}

這個方法還是很容易理解的，掃描brokerLiveTable這個集合裏面的所有Broker，看看每個Broker的最後一次註冊的時間。如果在2分鐘以前，那麼說明這個Broker已經掛掉了，應該移除這個Broker的信息。

四. DefaultRequestProcessor回調邏輯

Namesrv可以通過Netty響應其他進程的請求，比如Broker。那它的響應邏輯就是在DefaultRequestProcessor裏面，代碼如下：

public RemotingCommand proce***equest(ChannelHandlerContext ctx,
        RemotingCommand request) throws RemotingCommandException {
        ...
        log.info("request code: " + request.getCode());

        switch (request.getCode()) {
            case RequestCode.PUT_KV_CONFIG:
                return this.putKVConfig(ctx, request);
            case RequestCode.GET_KV_CONFIG:
                return this.getKVConfig(ctx, request);
            case RequestCode.DELETE_KV_CONFIG:
                return this.deleteKVConfig(ctx, request);
            case RequestCode.REGISTER_BROKER:
                Version brokerVersion = MQVersion.value2Version(request.getVersion());
                if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
                    return this.registerBrokerWithFilterServer(ctx, request);
                } else {
                    return this.registerBroker(ctx, request);
                }
            case RequestCode.UNREGISTER_BROKER:
                return this.unregisterBroker(ctx, request);
            case RequestCode.GET_ROUTEINTO_BY_TOPIC:
                return this.getRouteInfoByTopic(ctx, request);
            case RequestCode.GET_BROKER_CLUSTER_INFO:
                return this.getBrokerClusterInfo(ctx, request);
            case RequestCode.WIPE_WRITE_PERM_OF_BROKER:
                return this.wipeWritePermOfBroker(ctx, request);
            case RequestCode.GET_ALL_TOPIC_LIST_FROM_NAMESERVER:
                return getAllTopicListFromNameserver(ctx, request);
            case RequestCode.DELETE_TOPIC_IN_NAMESRV:
                return deleteTopicInNamesrv(ctx, request);
            case RequestCode.GET_KVLIST_BY_NAMESPACE:
                return this.getKVListByNamespace(ctx, request);
            case RequestCode.GET_TOPICS_BY_CLUSTER:
                return this.getTopicsByCluster(ctx, request);
            case RequestCode.GET_SYSTEM_TOPIC_LIST_FROM_NS:
                return this.getSystemTopicListFromNs(ctx, request);
            case RequestCode.GET_UNIT_TOPIC_LIST:
                return this.getUnitTopicList(ctx, request);
            case RequestCode.GET_HAS_UNIT_SUB_TOPIC_LIST:
                return this.getHasUnitSubTopicList(ctx, request);
            case RequestCode.GET_HAS_UNIT_SUB_UNUNIT_TOPIC_LIST:
                return this.getHasUnitSubUnUnitTopicList(ctx, request);
            case RequestCode.UPDATE_NAMESRV_CONFIG:
                return this.updateConfig(ctx, request);
            case RequestCode.GET_NAMESRV_CONFIG:
                return this.getConfig(ctx, request);
            default:
                break;
        }
        return null;
}

以Broker註冊爲例，RequestCode.REGISTER_BROKER （103）

this.registerBrokerWithFilterServer(ctx, request);

public RemotingCommand registerBrokerWithFilterServer(ChannelHandlerContext ctx, RemotingCommand request)
        throws RemotingCommandException {
        ....

        RegisterBrokerResult result = this.namesrvController.getRouteInfoManager().registerBroker(
            requestHeader.getClusterName(),
            requestHeader.getBrokerAddr(),
            requestHeader.getBrokerName(),
            requestHeader.getBrokerId(),
            requestHeader.getHaServerAddr(),
            registerBrokerBody.getTopicConfigSerializeWrapper(),
            registerBrokerBody.getFilterServerList(),
            ctx.channel());

              ...

                BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
                    new BrokerLiveInfo(
                        System.currentTimeMillis(),
                        topicConfigWrapper.getDataVersion(),
                        channel,
                        haServerAddr));     

       ...
}

public RegisterBrokerResult registerBroker(
        final String clusterName,
        final String brokerAddr,
        final String brokerName,
        final long brokerId,
        final String haServerAddr,
        final TopicConfigSerializeWrapper topicConfigWrapper,
        final List<String> filterServerList,
        final Channel channel) {

        ...

        if (null != topicConfigWrapper
                    && MixAll.MASTER_ID == brokerId) {
                    if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
                        || registerFirst) {
                        ConcurrentMap<String, TopicConfig> tcTable =
                            topicConfigWrapper.getTopicConfigTable();
                        if (tcTable != null) {
                            for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
                                this.createAndUpdateQueueData(brokerName, entry.getValue());
                            }
                        }
                    }
       }         

       ...
}

private boolean isBrokerTopicConfigChanged(final String brokerAddr, final DataVersion dataVersion) {
        BrokerLiveInfo prev = this.brokerLiveTable.get(brokerAddr);
        if (null == prev || !prev.getDataVersion().equals(dataVersion)) {
            return true;
        }

        return false;
}

如果是第一次註冊，或者dataVersion有變化，也就是isBrokerTopicConfigChanged返回true(Broker數據有變化，傳送過來的dataVersion就會+1)，那麼就會更新topicQueueTable集合，這個集合維護topic屬於哪個Broker的關係。

除了更新topicQueueTable之外，還會更新brokerLiveTable，顧名思義-broker的生命週期table

BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
                    new BrokerLiveInfo(
                        System.currentTimeMillis(),
                        topicConfigWrapper.getDataVersion(),
                        channel,
                        haServerAddr));

所以這也說明了一個問題，我們試用RocketMQ的時候，Topic不要太多，幾十幾百還是沒有問題，成千上萬就不好了。

46. 源代碼解讀-RocketMQ-namesrv

一. Namesrv作用

二. 流程圖

三. 源代碼流程分析

1. mqname.sh

2. NamesrvStartup

3. NamesrvController.initialize

4. RouteInfoManager.scanNotActiveBroker

四. DefaultRequestProcessor回調邏輯

[轉帖]使用NMT和pmap解決JVM資源泄漏問題原創

Python實現大麥網搶票的四大關鍵技術點解析

Python 安裝庫指令大全

salesforce零基礎學習（一百三十八）零碎知識點小總結（十）

一款開源的.NET程序集反編譯、編輯和調試神器

關於接口協議，你必須要知道這些！

【2024-05-21】以茶會友

70.Flutter筆記

67.Python學習筆記

24. iOS基本語法

66.Spring源代碼分析-代碼下載

65.iOS下面LKDatabase 導出和查看

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結