深度解讀暢捷通雲原生架構轉型實戰歷程

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9e/9ed0636491112f58908036369c599883.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"在信通院 2021 年雲原生產業大會上，暢捷通獲得 2021 年度雲原生優秀案例。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通公司是用友集團旗下的成員企業，專注於服務國內小微企業的財務和管理服務。一方面，暢捷通將自己的產品、業務、技術架構互聯網化；另一方面，暢捷通推出了暢捷通一站式雲服務平臺，面向小微企業提供以數智財稅、數智商業爲核心的平臺服務、應用服務、業務服務及數據增值服務，致力於建立“小微企業服務生態體系”。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據易觀國際的報告，目前暢捷通在國內小微企業雲服務市場的覆蓋率保持第一。有超過 130 萬家企業與機構通過使用暢捷通的軟件及服務，實現降本提效、持續創新、數智化轉型。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"早期，暢捷通的業務應用都是構建在公司自主研發的 CSP 平臺之上，包括客戶管家、易代賬、好會計和好生意等，每個企業分配一個獨立的虛機，資源完全隔離，初期該平臺方案極大簡化了開發的複雜度，提高了開發效率，滿足公司向雲服務發展的最初需求。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"新的需求","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着用戶規模的增多，現有爲每個用戶分配一個獨立虛機的方案產生的問題很突出。首先體現在虛機的數量日益龐大，在客戶轉換率不高的情況下，容易造成資源的浪費，同時運維成本也偏高；其次，不同的用戶軟件迭代的版本不盡相同，還需要專門設計維護多套版本的補丁更新系統；第三，產品需要停服上線，業務會出現短暫不可用的情況；最後，如果若干用戶的業務出現高峯期，不能快速彈性擴容，資源的利用率不高。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在此背景下，暢捷通研發團隊決定嘗試當前業界比較通用的雲原生架構設計方案，利用雲上的基礎設施，共享計算資源和存儲資源，採用容器化方案，快速彈性擴容，利用微服務架構，按應用更新產品，徹底解決當前運維成本偏高、彈性不足、資源利用不均衡的問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"小微企業的特點是數量多、單個企業業務量相對較小、企業 IT 能力有限。以暢捷通好生意產品爲例，採用現有的雲原生架構，能夠方便暢捷通快速開發出一款應對大規模小型用戶，支持彈性可擴展的 SaaS 應用。同時通過服務的編排，又能快速搭建出其他產品線，如智+。目前已經有超過 20 萬的付費用戶正在使用暢捷通提供的雲原生架構企業雲服務，每天產生的業務數據達百 G 以上。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"雲原生應用架構的驅動力","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"國內雲計算產品快速發展，企業應用往雲端遷移趨勢明顯，加上政府部門鼓勵企業上雲推出補貼政策，企業上雲已成爲大勢所趨。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"尤其在疫情階段下，商業模式的變革，消費方式的轉變，只有企業上雲才能更有利於推動企業加快數字化、智能化的轉型，更有效的幫助企業實現“客戶在線、業務在線、人員在線、管理在線”。而現在的雲原生技術帶來的價值能更好的幫助企業進行數智轉型。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1. 實時在線","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用智能接入網關，就近接入雲企業網，在 VPC 間、VPC 與本地數據中心間搭建私網通信通道，通過自動路由分發及學習，提高網絡的快速收斂和跨網絡通信的質量和安全性，保證用戶全球範圍實時在線，加速實現整個客羣的線上線下一體化。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2. 數智化","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過雲端廉價的存儲、強大的算力資源以及衆多的算法模型，暢捷通用極低的成本存儲海量數據並在線實時分析，爲用戶提供更多的商業決策。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3. 快速響應市場需求","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用 DDD 設計思想，利用微服務架構，快速開發高內聚、低耦合的應用。這些應用通過服務的編排，能快速組合更多的應用，滿足不同行業和領域的客戶羣體，達到快速上線、迭代優化的效果。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"4. 穩定高可靠","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"利用容器和微服務架構，可以快速構建和運行可彈性擴展的應用。系統出現故障或者性能瓶頸的時候，通過鏡像可以秒級恢復受損應用，保障了系統的高可用性。利用雲原生技術的紅利，暢捷通可以只關注業務的開發，一方面加速構建新應用，另一方面也可以優化現有應用並在雲原生架構中集成，達到奔跑中更換輪子的效果，去更方便地讓歷史存量的客戶升級到雲上。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"雲原生應用架構設計","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"雲原生應用架構設計路線","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原有產品是部署在物理 IDC 中，通過對 cloudfoundry 雲平臺的二開，實現每個租戶間虛機的隔離。但由於每個租戶獨享容器+數據庫，當用戶量達到幾十萬級時，數據庫的升級效率、容器部署成本、硬件運維複雜度都明顯提升。通過應用的微服務化、上雲來解決降本提效的問題迫在眉睫。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通通過基礎設施上雲、數據庫上雲、技術框架和業務框架的重構，實現了在多租戶之間容器部署、應用共享、DB 共享，產品基於 EDAS 及集成在其上的阿里雲容器服務 Kubernetes 版 ACK。希望通過雲原生的技術紅利，解決當前運維成本高、系統彈性不足、產品迭代交付週期長的問題。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"應用架構的改造","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1. 微服務架構","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將複雜應用按照業務的視角切分爲高內聚、低耦合模塊，這些模塊獨立開發、獨立發佈。業務領域一共分爲四層，即核心領域服務層、業務領域服務層、應用服務層和接口服務層。其中核心領域服務層包括授權、UOM、組織（Party）、產品、計價、促銷和存量模型模塊，主要提供核心領域知識、能力服務；業務領域服務層是提供好生意業務的業務功能，包括採購、庫存管理和銷售領域服務；應用服務層基於具體應用場景，調用領域服務，解決應用中具體的業務問題。每層服務都是一個單獨的微服務，基於 EDAS 進行服務的全生命週期管理，通過引入 Spring Cloud 實現服務的註冊發現以及治理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，由於 EDAS 無縫集成了 ACK ，支持以容器的形式託管應用到阿里雲 Kubernetes 集羣或混合雲集羣（其他雲域或IDC內自建集羣），因此能夠與暢捷通底層K8s集羣打通，實現了 K8s 集羣的定時彈性能力和基於 CPU/RT/Load 等某一監控指標的自動彈性能力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2. 數據一致性","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的單一服務架構，所有領域的功能都聚合在一個進程內運行，可以通過數據庫的事務來保證業務強一致性。但是暢捷通現在按分佈式微服務架構設計，不同領域模塊會建成獨立運行的微服務，微服務之間需要按照最終一致性方案來保證數據的一致。對於實時性要求高的場景，暢捷通採用 TCC 模型；對於實時性要求不高，可長過程處理的，暢捷通採用消息隊列的方式進行服務的解耦，達到最終一致性。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"技術架構的改造","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1. 容器化管理","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"核心應用部署在容器中，通過 Kubernetes 進行統一編排和運行調度。對於秒殺場景或者耗算力的異步任務，通過函數計算來按需構建。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2. 服務治理","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"引入微服務架構後，服務管理尤爲複雜，爲此暢捷通引入 Spring Cloud 一站式解決方案，使得開發只需要專注業務的發展，不去關注技術細節。通過 Spring Cloud 完成服務發現註冊、配置管理、限流降級、服務調用、數據監控等，實現降本提效，也降低了運維成本。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3. Gitops 流水線","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 Gitlab、Jenkins、Rundeck、K8s，搭建自研的 DevOps 流水線，按照微應用獨立構建、微服務自由組包、按照容器化進行發佈部署，保證了研發、測試、線上各階段的運行環境均保持不變。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"4. 數據庫層面的改造","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有租戶共享數據庫，按照應用、數據量等因素進行分庫分表；引入 OLAP 數據庫，分離交易庫和分析庫，避免大查詢拖累用戶的交易。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"雲原生應用技術框架","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/27/27147dfb9e378656193498456c0a6c1d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系統分爲前端展現層、業務中臺、技術平臺、運維中臺以及基礎設施層。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"前端展現層","attrs":{}},{"type":"text","text":"：使用基於微前端應用集成思想形成的 H5 前端開發框架，使用 qiankun 實現同構應用和異構應用的集成，支持多端開發。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"業務中臺","attrs":{}},{"type":"text","text":"：採用微服務架構的設計思想，基於 EDAS 平臺搭建而成，在應用服務層可以實現彈性伸縮、限流降級、流量監控等；業務模型通過元數據驅動並支持 GraphQL 查詢；利用 RocketMQ 實現了業務的解耦。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"技術中臺","attrs":{}},{"type":"text","text":"：包括容器管理、DevOps 流水線、微服務治理、鏈路追蹤等，是企業業務快速交付、系統穩定運行，業務快速創新的基石。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"基礎設施層","attrs":{}},{"type":"text","text":"：包括數據存儲層以及中間件。數據基於關係型數據庫存儲，如MySQL、PolarDB 等，通過 Tenant 標識在數據庫表級別實現多租戶共享數據庫；數據庫也支持讀寫分離，查詢操作只需要訪問讀數據庫即可。其餘中間件均由技術平臺進行了二次封裝，對業務屏蔽了具體的技術細節。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"前臺採用微前端框架","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/58/58b943814d9215ed01f09362e55d427a.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"**","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前端應用由於參與的人員增多、產品的應用模塊隨着需求不斷增加，已經從一個普通單體應用演變成一個巨石應用，好生意、智+、微商城等產品前端開發工程都在一起開發，相互影響，導致功能開發、迭代上線無法按照應用單獨部署。再加上前期業務和技術分層不清晰，導致公共組件抽象不夠，業務和技術強耦合，技術想單獨更新換代異常艱難。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通需要設計一套框架，確保暢捷通的業務代碼能平滑的遷移，且還能確保暢捷通能在不影響業務在線的情況下，進行技術的迭代更新。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通只需要在主系統構造一個足夠輕量的基座，再加上公共可複用的組件（包括基礎技術組件、基礎業務組件、公共業務服務等），然後讓各子應用按照共同的協議去實現即可。這個協議包括，主應用應該如何加載子應用，以及子應用如何被主應用感知、調度，應用之間如何通信等。同時暢捷通這個系統還應該有提供統一的 build 工程，讓各子應用不再關注配置、依賴、構建、發佈等操作。這樣，各個微應用之間就可以獨立發佈部署，不同應用之間的弱耦合，也降低了開發的複雜度。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"技術平臺-容器管理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"快速發佈環節：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於從 git 的源碼管理平臺和配置管理平臺的聯動，實現了容器鏡像的快速自動生成和管理，基於環境變量的區分和配置中心的統一管控，能實現一個容器鏡像多環境部署模式，並且能對每次的代碼進行安全和一致性掃描，保障代碼環節的安全穩定。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"閉環管理環節：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發佈到線上後，基於阿里雲 Prometheus 監控，異常信息發送到消息中心中，並且在消息中心數據匯聚和策略編排，形成了工單流的模式，實現有效數據的閉環管理。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"業務保障環節：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在容器彈性伸縮方面，暢捷通藉助 K8s 的 HPA 機制，基於阿里雲容器服務 ACK 最大化利用資源的能力以及業務層自定義指標，實現面對直播、秒殺、在線考試等突發流量下微服務的快速擴縮容。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"技術平臺-DevOps 流水線","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b3/b3eb2d4b0a5d38cef574fdc985529a9e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用管道方式，將原本獨立運行於單個或者多個節點的任務連接起來，實現單個任務難以完成的複雜流程編排。自動化構建、測試和發佈過程可輕鬆測試每次代碼更改並捕獲易於修復的錯誤。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過構建 DevOps 工具鏈，實現從需求下發、到代碼提交與編譯，測試與驗證到部署與運維的全過程支撐，打通軟件交付的完整路徑，提供軟件研發端到端支持。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"技術平臺-微服務治理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着業務的快速發展，暢捷通對原有的IT系統進行了大量的微服務化改造，以適應互聯網大型應用快速迭代以及頻繁發佈的需求。由於 SaaS 化企業管理雲服務具備用戶量大、業務複雜、調用鏈路長、與第三方應用系統深度集成等特點，給微服務化改造工作帶來了非常大的挑戰。暢捷通必須提升整體的微服務治理能力與監控能力，在頻繁的版本迭代中才能確保系統的穩定健壯。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d3a9cc6b0c276b9f8d94ee9040c39005.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"**最終暢捷通的微服務部署到阿里雲提供的企業級分佈式應用服務 EDAS 上，運行在 EDAS 上的 Spring Cloud 應用，可以享受到應用生命週期管理、無損下線、全鏈路流控等一系列針對微服務治理領域的能力增強。**特別在應用發佈的流程中，EDAS 所提供的平滑上下線以及灰度機制極大程度的提升了系統在版本更新期間的穩定性，降低了應用發佈所帶來的風險。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"技術平臺-全鏈路監控","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"海恩法則指出：每一起嚴重事故背後，必然有 29 次輕微事故和 300 起未遂先兆以及 1000 起事故隱患。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/59/59a93241fb1f8349748101998baeb3f7.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"尤其是暢捷通採用了微服務架構後，由於 SaaS 產品所涉及到的業務鏈路極爲複雜，當用戶反饋系統 Bug 或者性能存在問題之後，技術團隊需要耗費非常長的時間在錯綜複雜的鏈路之間定位故障源以及分析性能瓶頸。暢捷通也使用了一些 APM 工具類的產品，包括應用性能監控，用戶體驗監控，鏈路追蹤，問題診斷等，但是這類工具僅能定位框架級的問題，對於自定義函數以及進程中的異步處理均無法做到鏈路追蹤，在分佈式應用架構中這類 APM 發揮的作用就更少了。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"由於暢捷通的應用是託管在阿里雲的 EDAS 中，EDAS 集成了應用實時監控服務 ARMS","attrs":{}},{"type":"text","text":"，可以監控微服務的健康狀態和關鍵指標，並針對監控指標設置告警，及時發現並處理可能存在的異常或故障，以保障應用的健康和可用性，所以暢捷通只需要將業務操作與系統日誌、系統日誌和 ARMS 打通，就可以實現從業務出發，貫穿整個業務的生命週期，快速定位應用的性能瓶頸以及異常故障的位置。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲此，暢捷通實現了一套機制，將業務操作按照 Timeline 進行抽象建模，並結合系統日誌、阿里雲 ARMS 系統形成三位一體的全鏈路跟蹤機制。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原則上，除了讀操作之外的寫權限點所對應的交互都應當視作一次 BI。所以暢捷通可以簡單地認爲每個寫操作的權限點就是一個 BI。這就反過來要求後端提供的 REST api 必須是面向場景，每個API對應一個權限點。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BI 與 REST api 的 request_id 之間需要建立關聯，以便追蹤業務操作與系統日誌之間的關係。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"WebFilter 攔截所有 RESTful API的Request 和 Response ，獲取到請求和應答信息，通過交互協議中的取值公式從攔截到的數據中解析出具體的特徵、角色、關係等數據。在 Request 請求中增加 Create 操作將實體的原值自動記錄下來，在 Response 返回時額增加 Complete 操作，負責把新值寫入，並記錄前後變化。兩者在接口的調用開始和調用結束時配對使用。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在後臺日志裏面，記錄了 user_req_id 和阿里雲的 arms 的 uber-trace-ID 的對照關係。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過全鏈路跟蹤機制，暢捷通將業務的交互操作與阿里雲 ARMS 應用監控關聯起來，尤其是業務中還存在一些通過消息隊列進行解耦的操作，暢捷通都可以通過 BI 來進行追蹤，爲暢捷通的微服務體系更進一步的提供了監控能力。在接入 ARMS 之後，通過全鏈路信息排查以及應用實時診斷等工具，將定位系統故障源以及性能瓶頸的工作量降低到了之前的 50% 以下，極大程度的提升了 IT 團隊的工作效率。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"技術平臺-灰度發佈","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"灰度發佈（又名金絲雀發佈）是指在黑與白之間，能夠平滑過渡的一種發佈方式。在其上可以進行 A/B testing ，即讓一部分用戶繼續用產品特性 A ，一部分用戶開始用產品特性 B 。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"灰度期","attrs":{}},{"type":"text","text":"：新特性在灰度環境發佈一直到新特性部署到線上正式環境的這一段時間，稱爲灰度期。對於 2C 的應用，是以用戶作爲灰度的基本單位進行分流。而對於 2B 的應用，則是以租戶爲基本單位進行分流。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"灰度環境包括數據庫灰度和應用程序的灰度。若在數據庫層面支持灰度，則需要新建一個灰度 DB ，把參與灰度的客戶數據導入到灰度 DB 中，灰度結束後再把數據清洗合併到正式生產 DB 中。這個過程所需操作較多且成本較高，鑑於此，數據庫層面不考慮灰度。基於這個設定，需要遵循以下幾個約束：​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"灰度客戶的量控制在較小範圍，以儘可能縮小數據修復的範圍；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型設計嚴格遵從兼容原則，如：只增不減，字段避免複用；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於影響業務邏輯的系統數據需要有評估。比如增加了某個系統級的枚舉值，只對灰度客戶可見，可以考慮針對灰度客戶複製一份系統數據，灰度後刪除，系統後端代碼邏輯優先取租戶數據，取不到再獲取系統級數據。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"應用程序的灰度因爲後端對外提供的服務都是 REST API ，可以採用支持調用外部服務的負載均衡或反向代理獲取灰度用戶名單後，對用戶進行分流。後端接口的兼容性需要保證，當無法保證兼容性的情況下，需要強制前端一同升級。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f85b10d84884f138488083ecb0aa41ce.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"基礎設施層-數據庫讀寫分離","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"初期雲服務上線的時候，用戶規模小，數據量也小，所以線上環境 MySQL 沒有實現讀寫分離方案。在平穩運行一段時間，隨着用戶規模導致 PV 和 UV 增加，給數據庫帶來了巨大壓力，主要體現在如下三點：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"複雜業務，多表聯查效率低下，導致業務功能響應不夠及時甚至查詢超時;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務高峯期傳統的數據庫無法快速升配，只能等待業務高峯過去或者做流控;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務代碼對主從延遲敏感，無法充分利用從庫。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"初期採用 Sharding-JDBC 實現讀寫分離，但現有產品讀寫操作互繞，不易將讀操作分離，且讀庫和寫庫的同步延遲較大，不太滿足業務需要。調研 PolarDB 後發現，其集羣地址直接支持讀寫分離，且與 MySQL 語法 100% 的兼容，對業務無影響。通過 PolarDB 的分鐘級彈升能力，能快速升配；其特有的並行查詢能力也可以降低複雜查詢的響應時間，同時提升系統的併發能力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以暢捷通將數據庫從 MySQL 遷移到 PolarDB，並採用了一寫多讀的模式，並對耗時的報表查詢業務單獨指定讀節點，降低了耗時操作對其他業務的影響，保障了用戶的正常交易操作。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"運維中臺-監控體系","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的報警機制，以消息爲載體，儘量簡潔，更傾向於報警即故障模式，涵蓋信息少，對後期判斷往往起到一個提醒開始的狀態。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着運維模式的提高，大量高效工具的結合，自愈，時序事件等相關係統不斷湧現，同時近年來興起的雲原生模式，將實現智能預警的老大難問題：信息標準化問題，給出了優質的解決方案，也加速了升級監控結構的步伐，報警的單一模式已不再適應需求，運維人員更希望報警有意義，有過程、有結論。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通長期致力於以客戶體驗爲基礎的立體化監控架構，從客戶性能與體驗損失的緯度，對監控信息分級，分權重，通過業務鏈中各環節的客戶畫像，特徵模型，精準監控用戶體驗。可在用戶未感知的情況下預警並進入處理流程，進入流程的事件在內部消息中心扭轉，關聯計算，分析根因以及自愈方案。並根據事件處理模型響應事件，發送報警，報警涵蓋事件的關鍵信息與結論。在提高了預警有效行的同時，也避免的傳統故障處理模型中的各種效率損失。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"雲原生應用服務特點","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"DevOps 工具-故障檢修中心","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"衆所周知，分佈式系統遵從 CAP 理論，暢捷通通常選擇滿足 AP 而放棄 C（一致性）。按照墨菲定律，暢捷通最終是需要通過人工干預來保證數據的最終一致性。那麼檢查發現不一致的數據、區別錯誤場景並按規程修復，就需要有一套系統來輔助進行。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不同於系統運維，業務檢修面向的是業務和服務，通過租戶和租戶數據的監控來發現和解決業務問題。系統運行中，除數據不一致之外，還有其他一些常見故障，也需要爲支持和解決一些較爲普遍有共性的客戶問題提供便利，以及提供部分分析視圖幫助研發或運維掌握瞭解租戶狀態和趨勢，這些訴求的響應也都會因爲生產環境的物理隔離和安全性要求，而變得低效，只有通過固化到系統中形成工具集才能更好地解決。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以，暢捷通提供了一套業務檢修系統，負責業務數據的監控、問題診斷，故障修復、態勢預警等。整體設計如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/94/9430d7a94c4b39c04e7d611c84280fd6.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前系統針對開通失敗、定時任務失敗、死信、數據稽覈違規、導入超時、TCC 失敗六類故障進行定時檢查，並與運維的 Midas 消息系統集成，及時發現問題，並向具體負責人發出釘釘告警；負責人通過故障展示的看板和明細列表來查看具體原因；針對具體故障，還提供查看上下文日誌、重新開通、死信重放、數量帳重算、重提交等輔助手段解決故障。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"安全服務-內容安全","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通的產品會涉及到協同中的聊天信息，電商類的商品評價，這些內容可能會被外人惡意使用，將一些非法廣告、網絡詐騙、恐怖信息等內容錄入暢捷通的產品中並傳播出去，所以暢捷通的運維安全部門需要增加內容安全的監控，保障公司產品的純淨、健康運行，避免網絡犯罪行爲發生在暢捷通公司的產品和用戶之中。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8a/8ad97d987850025d60c76f99c5b8a519.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通借用系統的操作日誌和消息隊列，將內容安全檢測和業務功能進行解耦，藉助函數計算的能力，按需調用雲服務廠商提供的安全檢測能力（文字類、圖片類等），去識別可分享的業務，內容是否符合安全法，檢測結果還增加了人工審覈及誤判赦免。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"數用分離","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6a/6a8574d85efe4fcea9e81810c9b9bdb4.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"統計分析類數據，暢捷通每天通過 DTS 將原始數據從關係型數據庫增量同步到數據倉庫裏，在固定時間進行數據清洗、彙總統計分析後，再將這部分數據傳輸 Elasticsearch 。同時暢捷通也利用 MaxCompute 的多任務計算能力，進行業務數據的標籤計算，計算結果也傳遞給 Elasticsearch ，業務系統通過 Elasticsearch 的 REST API 訪問這些數據，並展現給最終用戶。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"全鏈路流量控制技術+端雲聯調","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於微服務間的依賴，開發人員無法在本地完成開發和測試，必須依賴一套測試開發環境。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"研發效率方面","attrs":{}},{"type":"text","text":"：當上百人的開發人員共享一套環境，開發態的代碼頻繁變更，質量較低，服務之間相互影響，開發環境經常中斷，調試困難，嚴重影響了開發效率，研發希望每個項目能提供一套環境，如圖：​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3f/3f1c1213588c7030bf2f49d829315116.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"研發成本方面","attrs":{}},{"type":"text","text":"：如果爲每個項目提供全量環境，計算資源成本很高，運維成本也會激增。（在開發態有近百個微服務，按 20 個項目並行開發計算，需要近 2000 個POD/ECS 的計算資源成本和運維成本）產品在成長期，並行的項目和服務數都在增加。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合效率和成本方面的因素，暢捷通引入網關、全鏈路流量精確控制技術、端雲聯調技術，用基線環境+增量修改的應用疊加出項目開發環境：在入口處根據企業 Id 進行流量打標（根據企業 ID 在請求中注入環境標識），如：https://cloud.chanjet.com/req?orgId =企業 ID ，環境標識隨請求在整個執行流程中流轉（http 請求，rpc 請求，定時任務，消息等），通過環境標識，對微服務調用/消息進行精確控制。如圖：​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33ceb5aa298338a2de120aa794505941.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通在基線環境部署最新上線的代碼，在項目環境中只部署涉及修改的應用：​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（1）爲項目研發提供穩定的獨立項目環境，使研發能按“小規模，多批次”（DevOps 最佳實踐）的方式快速協作；（2）節省了資源、降低了運維成本。（按每個項目修改 5% 的應用，在開發態有近百個微服務，按 20 個項目並行開發計算，將節省 2000*95%=1900 個ECS/POD）；（3）使用端雲聯調技術，方便開發本地PC加入到項目環境調試。如圖所示：​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ef/efc28c807e834f7e1f53931cb4fda4f5.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"雲原生帶來的技術價值","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"高彈性可擴展","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"系統可以在應用服務層和數據庫層支持橫向擴展。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"**應用方面：**通過微服務架構，容器化部署，技術平臺可以感知服務器負載，彈性伸縮服務節點，實現集羣擴/縮容，快速補充計算能力。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"**數據庫方面：**數據庫支持快速擴容讀節點，以支持更多併發請求，同時租戶實現數據隔離，並可根據負載實現跨庫遷移。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"微服務化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在微服務架構和設計階段：​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"引入 MDD 領域驅動方法論進行應用架構設計和領域模型設計；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按照微服務的設計原則進行服務的拆分和職責、關係定義，確定服務接口和領域模型；","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲一個複雜的 ToB 應用，暢捷通按照如下原則進行微服務的拆分：​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可用 - 拆分的模塊能夠滿足應用的需要；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好用 - 拆分的模塊能夠通過比較簡單、清晰的方式組成應用，服務應用價值；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"美 - 用最簡單的方式表達複雜問題；業務人員容易理解。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Back-end For Front-end","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務器端採用微服務架構之後，暢捷通需要在前後端之間增加 BFF 層，簡化前後端人員的協同開發複雜度。BFF 層基於 Node 實現，暢捷通選擇了 Egg.js ，並基於Egg.js 做了分層封裝，在 Node 自身技術升級的情況下可以不影響業務開發。爲了提高效率，暢捷通還在 Node 層接入了緩存中間件 Redis ，並利用 GraphQL 做聚合查詢。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"端雲聯調方案","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採用微服務架構後，每套環境部署需要 30+ 的微服務容器，如果開發階段存在多特性並行，爲每個特性分支單獨部署環境，資源消耗太大。爲此，暢捷通引入了網關、全鏈路流量打標控制技術、端雲聯調技術，用基線環境+增量修改的應用疊加出項目開發環境，以最小代價快速拉起一套完整的開發調試環境，降低了開發成本。同時開發人員也可以將本地電腦直接註冊到微服務環境中去，在本地完成上下游業務的驗證和測試，極大地提高了開發人員的工作效率以及提交代碼的質量。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"離線數據分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"早期，暢捷通的數據庫只有 OLTP ，數據存儲在 MySQL 裏，隨着數據量的增多，統計分析類的查詢不僅自身查詢時間長，還消耗了在線交易庫的資源，影響在線交易的響應時間。僅僅通過雲原生數據庫 PolarDB ，進行讀寫分離，只能緩解一部分查詢壓力。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對系統中一些實時性要求不高的分析，暢捷通引入了數據倉庫進行離線數據的處理，通過 DataWorks 工具進行 ETL 和統一調度，在 MaxCompute 中進行大數據指標計算和標籤計算。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"全鏈路灰度方案","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"暢捷通的系統，實現了從前端到後端服務、消息隊列、定時任務的全鏈路灰度。前端靜態資源：區分灰度靜態資源和正式環境靜態資源。登錄後由登錄信息裏的租戶，確定應該路由到哪個環境。消息隊列：區分灰度消息隊列和正式消息隊列。根據環境變量進行消費定時任務：定時任務的任務執行方，通過環境變量和租戶名單，來決定是否執行。後端 Rest 接口：在 Nginx 中通過 lua 腳本，解析 url 路徑，根據灰度名單中的租戶信息進行路由。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"雲原生帶來的業務價值","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"容器化運維管理：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 EDAS+ACK 模式的容器化部署和管理，實現了應用發佈的提升，並且在彈性伸縮和容量管理，也可以加快異常識別，故障自愈。並且在 2（異常識別）-5（快速定位）-10（自愈止損）的目標上不斷攀升。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"可觀測性：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於日誌平臺和數據中臺的結合分析，實現用戶畫像和用戶軌跡的展示，並且對數據的不斷挖掘，可以實現用戶體驗百分數量化，產品質量百分數量化，從而在服務團隊對接上提供數字化保障。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"成本方面：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從 IDC 機房容器模式到雲平臺容器化的升級，不必要投入到硬件設備的大額投入和替換設備上，從而在設備成本和人力成本上大幅度節省了很多，並且新的需求和想法都可以按量投入，彈性伸縮。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"DevOps：","attrs":{}},{"type":"text","text":"對接雲原生環境的 devops ，在研發需求支持上，節省人員 50%，構建及部署效率上提升了 4 倍。其中 2020 全年更新次數 12734 次，成功率 91.8%。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"多環境資源：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過多特性灰度方案的模式，7 套開發環境合併爲1套。其中每套環境 90 個微服務，可以實現靈活性的動態調整和擴展。節省 450 個節點。​","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雲原生容器化**：**容器化 EDAS+K8s 部署模式，可以實現節點的彈性伸縮，特別是支撐了暢捷通直播活動，秒殺場景。不僅僅節省了人力的部署等環節投入，也節省了秒級伸縮的成本。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"多雲平臺：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過 DevOps 和容器化模式的建立，滿足了用戶多雲場景的需求，並且節省人力50%，實現多雲環境的自動化運維模式。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"數據庫升級：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好生意從 MySQL 升級到 PolarDB 後，其中處理性能提升 20%~40% ，併成功的將無法支持的低端手持 pda 適配工作成功復活。抽樣回訪客戶，滿意度也大幅提升。​","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"穩定性：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"​基於雲原生基礎組建的對接，雲產品可以實現 5 個 9 的 SLA 服務，這樣避免了自身搭建開源組件的穩定性和安全風險。並且在組件功能上也能享受到各種業務需求功能的福利。對接雲原生後，可以提出 2（告警）-5（定位）-10（止損）的穩定性保障目標，從而在用戶滿意度上不斷提升。","attrs":{}}]}]}