消息隊列上雲挑戰與方案：騰訊雲的 Apache Pulsar 實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雲原生大潮風起雲湧，企業不再停留在理念層面，目前已在多方面落地。消息隊列作爲關鍵技術基礎設施之一，也在雲原生時代面臨着挑戰和機會。本文從傳統消息隊列上雲所面臨的三大挑戰說起，並以 Apache Pulsar 爲技術案例，深入淺出地講解了如何打造適配雲原生的消息隊列。希望本文能對大家提供參考。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1 背景介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如今，雲原生的概念已經滲透到了軟件開發的方方面面。雲原生不再只是未來的設想，而是一個現在進行時。開發人員在開發設計之初就需要考慮未來如何在雲原生環境上部署、運行服務，即如何“上雲”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在雲上，消息隊列將成爲一種基礎設施，像自來水一樣，可以隨時按需使用，並且有無限容量。用戶無需關心消息隊列的型號、規格，或是否需要升級配置，只需專注上層業務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"騰訊雲是騰訊集團傾力打造的雲計算品牌，面向全世界各個國家和地區的政府機構、企業組織和個人開發者，提供全球領先的雲計算、大數據、人工智能等技術產品與服務，助力各行各業實現數字化升級。爲了更好地爲廣大用戶服務，提供金融級可靠消息服務，騰訊雲開啓了消息隊列上雲之路，目前 Apache Pulsar 在騰訊雲上已經大規模使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2 傳統消息隊列上雲遇到的挑戰"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列在上雲過程中遇到了很多新的挑戰，比如如何平滑擴容、如何管理海量分區、如何保證異地多活等高可用性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"挑戰一：平滑擴容能力不足"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統消息隊列（如 Kafka 等），在平滑擴容方面存在很多不足，很難做到快速、無感知地擴容。由於 Kafka 的數據保存在 Broker 上，而每個 Broker 上的分區數據是有狀態的，因此每個 Broker 上數據可能不同，客戶不能通過簡單地增加 Broker 數量完成擴容。當集羣增加新 Broker 進行擴容時，會涉及數據遷移和同步，進而引發磁盤 IO 和網絡消耗。當流量突然爆發，業務本身使用 IO 和網絡帶寬就會很高。此時，如果由於集羣容量不足而觸發擴容，遷移的帶寬佔用和數據冷讀會直接影響到上層業務的使用，造成高延遲和錯誤率飆升。如果寫入的速度比遷移速度更快，那可能永遠都無法完成數據遷移。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，通常 20% 的客戶佔用了 80% 的流量，有些用戶對底層消息隊列並不熟悉，使用方式可能不規範，導致 Broker 端出現數據傾斜，很容易造成某些節點的磁盤佔滿，而其他節點的磁盤有比較大的空間。只採用簡單擴容、遷移數據的方式根本無法解決這種情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"挑戰二：海量分區管理無法承載"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有些用戶的業務比較特殊，單個分區的流量不大，但總體使用的分區數很多。現有的一些消息隊列，很難承載海量的分區，例如：如果一個 Kafka 集羣分區數太多，當出現 Leader 或者 Controller 切換時，恢復時間會很長。另外，如果不使用 SSD，文件寫入變得分散，可能出現複製跟不上，導致 ISR（In-Sync Replicas，副本同步）頻繁波動等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決上面這個問題，通常我們會爲每個用戶部署一個集羣，保證每個集羣的分區數不會過多。但由於每個集羣總流量不大，會造成集羣資源使用率不高，有大量閒置資源，無法發揮雲原生環境的優勢。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種情況下，最理想的方式是多個用戶共用一個集羣的資源，分別限制資源的使用量，避免出現資源浪費。但如果把所有用戶全部聚集起來放在同一個集羣運行，又可能會出現幾十萬、上百萬分區數問題，這就是我們的第二個挑戰——海量分區管理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"挑戰三：無開箱即用的異地多活解決方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"很多金融級用戶的業務場景對高可用、RTO（Recovery Time Objective，業務恢復時間）等指標要求非常高，因此需要同城多機房或者兩地三中心的異地多活的方式。對於強一致的異地多活，現有的消息隊列很少有提供開箱即用的完整方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3 遇見 Apache Pulsar"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果使用傳統的消息隊列上雲，要解決上述問題需要費一番功夫。通過調研，我們發現爲雲原生打造的下一代分佈式消息系統 Apache Pulsar 能很好地解決上述的大部分問題。下面針對上述各種挑戰，我們從 Apache Pulsar 具備的能力做下針對性概述。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"支持秒級平滑擴容"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar 支持雲原生環境，可以充分利用雲原生環境的彈性能力，達到自動、無感知的擴容，按需使用，不影響上層業務。Apache Pulsar 使用 BookKeeper 作爲數據存儲層，而 BookKeeper 原生避免數據傾斜問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar 上層 Broker 無狀態，原生支持平滑擴容。當流量突發增加時，只需要增加一個 Broker，然後等待部分 Topic 重新分配到新的 Broker 上，流量就會自動遷移到新的 Broker 上。整個過程只涉及元數據的修改和 Topic 分配的計算，實現秒級自動遷移。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"具備海量分區支撐能力"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在雲上，海量分區是常見問題。假設客戶有 100 萬個分區，如果把每個分區的元數據信息都保存起來，那總體數據量會有幾百 MB，光是下載這麼多的數據都需要很多時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar 沒有完全解決所有問題，但已經具備支持海量分區的能力。Apache Pulsar 抽象了 Bundle 的概念。Bundle 的元數據保存在 ZooKeeper。每個分區落在哪個 Broker 上，這種關係信息不會直接保存起來。Apache Pulsar 使用一致性哈希，把 Bundle 作爲哈希環中的節點，讓所有的分區散列上去。我們只需保存 Bundle 與 Broker 之間映射關係的信息即可；分區與 Bundle 之間的關係是固定的，可以通過散列動態計算出來，不需要保存每條關係。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/a8\/a8061a9b2c4cd38d6a782c56044282f8.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如此一來，我們需要存儲的元數據就有幾個量級的下降。在切換 Broker 時，基於一致性哈希的優勢，分區再平衡只會涉及到部分變動，可以迅速重新進行分配。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4 Apache Pulsar 在騰訊雲上的實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過調研後，我們決定基於 Apache Pulsar 打造一款新的消息隊列——TDMQ，開啓 Pulsar 在騰訊雲上的實踐之路。下面和大家分享下 Apache Pulsar 在騰訊雲上的實踐經驗，探究 Pulsar 如何快速適配雲原生環境"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"雲原生下的平滑擴容"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們利用 Apache Pulsar 支持雲原生環境進行平滑擴容。常見的擴容場景分爲幾種情況："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a) 當分區數量遠大於 Broker 數量時，新增 Broker，分區用一致性哈希（hash）方式自動遷移 Topic 到新的 Broker，然後新的 Broker 就可以對外提供服務，分擔壓力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b) 當分區數量少於 Broker 數量時，增加分區，讓流量分佈到更多 Broker 上，從而實現平滑擴容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/9e\/9e31b734643ba9edfef4dcd6d695ae1c.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"數據傾斜與存儲層擴容"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Broker 不保存數據，Broker 通過 BookKeeper Client 把數據保持到 BookKeeper 集羣中，每個節點我們稱爲 Bookie。最終還是無法避免 Bookie 是有狀態的，問題又回到了原點，有狀態的 Bookie 又如何實現平滑擴容呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，我們需要了解 Apache Pulsar 的存儲機制。Pulsar 使用 Quorum 機制來保證數據的一致性和高可用。當 Pulsar 持久化一條消息時，Broker 使用 BookKeeper client 同時並行寫入多個 Bookie 節點，根據消息的 Ack 數，來判斷有多少數據寫入成功。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ENSEMBLE SIZE（E）：可用 Bookie 的數量"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"WRITE QUORUM SIZE （QW）：並行寫入消息的 Bookie 數量"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"QUORUM SIZE （QA）：Ack 消息的數量"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/6f\/6fad2ccf17293682bb8222d90c3d677c.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果 Bookie 的數量大於 QW 的值，數據會以條帶化的方式落到不同節點上，過程如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/34\/34ce6310143026842552f8502d6be8cd.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一條消息，Broker 會同時寫入 Bookie1、Bookie2、Bookie3。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二條消息，Broker 會同時寫入 Bookie2、Bookie3、Bookie4。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三條消息，Broker 會同時寫入 Bookie3、Bookie4、Bookie5。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第四條消息，Broker 會同時寫入 Bookie4、Bookie5、Bookie6。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"…"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種條帶化寫入的好處顯而易見，既能充分利用每個磁盤上的 IO，還能讓數據存儲近似均勻，避免出現數據傾斜問題。當增加新 Bookie 節點後，無需等待數據遷移就可以對外提供服務，整個過程非常平滑。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"海量分區：降低和去除 ZK 依賴"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"海量分區是我們要解決的一個重要問題。如上所述，Pulsar 抽象的 Bundle 的概念能很好地幫助我們解決上雲過程中遇到的海量分區管理問題。但 ledger、cursor 等其他元數據還是存儲在 ZooKeeper 中。當分區數量達到百萬時，ZooKeeper 會不堪重負。因此，我們基於 Pulsar 做了一些改進——降低和去除 Broker 對 ZooKeeper 的依賴。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Apache Pulsar Broker 對 ZooKeeper 的依賴主要包括以下幾個方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"負載狀況"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"生產、消費"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Policy 存儲"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Leader 節點選舉"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過對依賴的分析和梳理，我們確定了降低和去除對 ZooKeeper 依賴的兩步目標："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"a. 降低 Broker 對 ZooKeeper 的依賴。即使 ZooKeeper 掛掉了，Broker 至少在一段時間內不會受到很大的影響，可以繼續提供讀寫服務，等待 ZooKeeper 的恢復。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"b. 完全去除對 ZooKeeper 的依賴。將元數據全部保存到 Bookie，不再依賴 ZooKeeper 存儲數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體地，我們對代碼做了一些改動，如："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 ZooKeeper 掛掉、本地緩存超時的情況下，讓本地的緩存快照不過期，Broker 保持當前元數據不變，繼續使用緩存中的元數據。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在寫入過程中，如果 Ledger 承載的 entry 數量已經超過了限制的大小，Apache Pulsar 會關閉當前 Ledger 並重開一個 Ledger。我們優化後，在這種情況下，Broker 暫時不關閉當前 Ledger ，而是繼續寫入，從而避免訪問  ZooKeeper。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用外部存儲保存負載狀況等數據。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"只在啓動時依賴 ZooKeeper 選舉 Leader 節點，後續則通過訂閱的 Exclusive 特性實現選主。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"增強原生異地複製能力，提供強一致方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有些用戶的業務場景對高可用的要求非常高，需要同城多機房或者兩地三中心的異地多活。Pulsar 本身有相應的異地複製能力，但屬於異步複製。異步複製就要看用戶對 RTO、RPO（Recovery Point Objective，恢復點目標）的容忍程度，如果要求 100% 可靠，一條消息都不能丟，那異步複製不能滿足要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，異步複製即使爲全量複製，爲了保證消息順序，同一時間我們通常只會使用一箇中心，這樣整體資源利用率最多爲 50%，資源使用率不高。另外，無法保證災備中心何時能接管流量，接管後所有業務是否正常運行。因此，我們提供了強一致的方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/ac\/acab5b76d03f089b9e9a06140f5962cf.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1. 同城跨機房"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Broker 跨機房。多數據中心之間地位均等。正常模式下，多數據中心協同工作，並行爲業務提供訪問服務，充分利用資源。一個數據中心發生故障或災難，其他數據中心正常運行，立即對關鍵業務或者全部業務實現接管，達到互備效果，使上層業務不會有明顯的感知。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/ae\/ae2b24ae265d7346e2a5975ad0d69d63.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2.三中心高可用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"跨地域場景，分爲生產中心、同城容災中心、異地容災中心。當雙中心由於自然災害等原因而發生故障時，異地災備中心可以用備份數據恢復業務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/b6\/b673981c95394892bcfb6b1b0b49ed44.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無論是同城跨機房還是三中心的高可用，對寫入都有限制——限制每條消息必須要跨機房 \/ 地域寫入，Ack 成功纔算成功。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，我們實現了自動無感知切換模式。如果找不到相應的備份地域，自動通過模式切換進行降級，使用戶可以在單個區域裏面繼續寫入、生產、消費，實現無感知切換。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Pulsar 具備跨機架、跨區域感知的能力，雲上的運營端配合 Broker 中"},{"type":"codeinline","content":[{"type":"text","text":"bookkeeperClientRackawarePolicyEnabled"}]},{"type":"text","text":"、"},{"type":"codeinline","content":[{"type":"text","text":"bookkeeperClientRegionawarePolicyEnabled"}]},{"type":"text","text":"、"},{"type":"codeinline","content":[{"type":"text","text":"bookkeeperClientSecondaryIsolationGroups"}]},{"type":"text","text":" 等參數，就能實現上述能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了實現 Broker 的高可用，我們也實現了跨機房 \/ 地域 ZooKeeper 的高可用。爲了能夠快速恢復集羣，每一個數據中心的 ZooKeeper 都有 Observer。Observer 不參與投票和選舉。當集羣節點數不足時，Observer 可快速切換爲對應的 Follower ，參與選舉，保證  ZooKeeper  集羣的高可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，我們弱化了對  ZooKeeper  的依賴。即使一段時間內 ZooKeeper 不可用，Broker 還可以繼續對外提供服務，弱化上層業務對切換的感知。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於地域之間網絡延遲，保證強一致性跨地域容災，會讓上層業務使用消息隊列的延遲上升，因此需要業務方根據自身的實際情況，權衡選擇具體的異地多活方式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5 未來寄語"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相對於其他傳統消息隊列，Apache Pulsar 藉助存儲與計算分離的雲原生架構，以及支撐平滑遷移、承載海量分區、跨區域數據複製等原生功能特性，成爲解決原有消息隊列上雲挑戰的最佳解決方案之一，目前不斷吸引衆多國內外企業落地並聚集了衆多活躍開發者。但當前 Apache Pulsar 距離最終的 Serverless 化、無規格化、無限流量等願景目標還有一段路要走，這也正是 Apache Pulsar 社區衆多貢獻者的價值所在，期待大家能夠參與進來。我們會繼續積極和社區合作，一起完善 Apache Pulsar 生態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雲原生是正在發生的重要技術事實，雲原生落地之路也剛剛開始，期待通過豐富的場景和實踐，持續推動雲原生前進。希望上述我們在消息隊列方向的探索與實踐能夠帶給有着類似需求的同行一些參考。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在騰訊雲上，我們基於 Apache Pulsar  推出了 TDMQ 消息隊列，除了上述改動，還有其他新特性。2021 年，TDMQ 會提供更多的新特性，支持 HTTP、AMQP 等多種協議，方便使用其他消息隊列的用戶無縫遷移到騰訊雲上。同時，我們還會有更多增強能力，如  Function、金融級 SLA 承諾、全球消息同步、Serverless Topic、彈性用量等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"林琳，Apache Pulsar PMC 成員、騰訊雲專家工程師，專注於中間件領域，在消息隊列和微服務方向具有豐富的經驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}