非HTTP應用或批處理應用如何進行全鏈路監控

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在近期的應用性能問題分析中, 有時候會發現業務異常, 但是卻無法在APM監控中找到相關的請求, 這可能是由於請求並非HTTP請求, 而可能是批處理或其他非HTTP請求導致的. 因此無法監控分析."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外, 有些時候, 我們想要了解這類應用的:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● TCP解析 處理性能;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 批處理中SQL性能..."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼就需要對非HTTP應用或批處理應用如何進行全鏈路監控. 下面介紹如何實現."}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"具體實現概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來具體是通過Dynatrace AppMon 來實現對非HTTP應用或批處理應用的全鏈路監控."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"監控前後的效果對比如下:"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/52/5240abf71c079563f8d8f42de45b7e9c.webp","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 未監控之前: 看不到一條事務"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 監控後: 事務數鉅細無遺"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體的實現是利用了Dynatrace的"},{"type":"text","marks":[{"type":"strong"}],"text":"Entry Points"},{"type":"text","text":"(入口點)功能."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個 "},{"type":"text","marks":[{"type":"italic"}],"text":"entry point"},{"type":"text","text":" 就是一個新的PurePath(Dynatrace 裏的名詞, 可以理解爲一條事務)的開始. 通過適當的配置, 放置一個適當位置的, 活動的sensor(傳感器, Dynatrace監控的微探針)來實現. 對於基本用戶, 自帶的Sensor Packs(傳感器包) (如: Servlet, WebService, ASP.NET... 它通過這些實現對HTTP協議請求的監控)就已經夠用了."}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"具體實現步驟"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果一個事務沒有被抓取到(比如: mule的tcp請求, 批處理..), 那麼就需要以下操作步驟:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. (可選)做CPU採樣, 找到"},{"type":"text","marks":[{"type":"italic"}],"text":"入口點"},{"type":"text","text":" 方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 配置指定方法的傳感器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 熱部署或重啓Agent生效"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"找到入口點方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你是一名開發, 且對你要監控的程序的 "},{"type":"text","marks":[{"type":"italic"}],"text":"入口點"},{"type":"text","text":" 方法瞭如指掌, 那麼只需要提供這個方法的完整名稱就可以了. 比如:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● TCP請求的入口點可能是這樣的: "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#F5222D","name":"red"}}],"text":"com.xxxx.ap26.getaway.transport.tcp.protocol.AbstractTcpHeader"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● Batch Job的入口點可能是這樣的: "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#F5222D","name":"red"}}],"text":"org.*spring*framework.batch.thread.run"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你不清楚"},{"type":"text","marks":[{"type":"italic"}],"text":"入口點"},{"type":"text","text":"方法, 可以通過"},{"type":"text","marks":[{"type":"strong"}],"text":"CPU採樣"},{"type":"text","text":"的方式進行分析."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU採樣可以分析當前運行的的Java或 .NET程序的線程堆棧, 並找出是否以某一個方法長時間運行或被頻繁調用.分析流程如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. 點擊Dynatrace上的"},{"type":"text","marks":[{"type":"strong"}],"text":"CPU 採樣儀表圖"},{"type":"text","text":". 如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/15/156fe8cb687ac81ef2180189f2d32d08.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 然後選擇你要監控的應用Agent, 點擊確定進行採樣. 大約2分鐘後停止採樣."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/22/220b0de009358e0be04bc5d9a5740c22.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3. 採樣完成後, 對樣本進行分析. 如下圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"   分析起來其實挺簡單. 我們要找的 "},{"type":"text","marks":[{"type":"italic"}],"text":"入口點"},{"type":"text","text":" 方法一般滿足這麼幾個條件:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" a. 位於線程棧的較底層, 但是肯定在線程啓動方法(如: "},{"type":"codeinline","content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#F5222D","name":"red"}}],"text":"Thread.run"}]},{"type":"text","text":")之後"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" b. 被頻繁調用(如果是tcp請求這類的話)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" c. 整個線程執行時間較長或CPU時間較長 (如批處理, 或tcp處理報文)"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8d/8dc9c81c74dd71137590ad12f1d11ba4.webp","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4. 根據以上條件, 我們會找到那個 "},{"type":"text","marks":[{"type":"italic"}],"text":"入口點方法"},{"type":"text","text":", 這個方法應該和大神開發給出來的是一樣的, 比如就是:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#F5222D","name":"red"}}],"text":"com.xxxx.ap26.getaway.transport.tcp.protocol.AbstractTcpHeader"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"配置該方法的傳感器"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"找到該方法後, 直接在上圖裏選中該方法, 點擊右鍵, 會彈出一系列菜單, 選擇: "},{"type":"text","marks":[{"type":"strong"}],"text":"添加這個方法爲傳感器"},{"type":"text","text":". 然後會彈出如下對話框:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"勾選紅框中的: "},{"type":"text","marks":[{"type":"strong"}],"text":"Allow to start PurePaths from this method (Entry Point)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置好之後點擊確定."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9d/9d6488a52ee3b1d4dd8541e043cb2954.webp","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼生效後, 就會對該方法進行字節碼注入, 並且, 只要捕獲到該方法, 就會認爲該方法是一條事務的"},{"type":"text","marks":[{"type":"italic"}],"text":"入口點"},{"type":"text","text":", 後續該事務可能會:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 調用其他任何方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 調用Web Service"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 執行SQL"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 拋出 Exception"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● 輸出 Error log"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ● ..."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"都會被完整的記錄, 記錄響應時間, 成功/失敗, 相關參數, web service 請求, sql語句, 異常堆棧, log日誌...應有盡有."}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"熱部署或重啓Agent生效"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然, 還是要使剛剛添加的方法監控規則生效纔會有一切. 要使之生效, 有2種辦法:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1. Dynatrace Agent 熱部署 (SUN JDK 1.6+)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. 重啓Dynatrace Agent (IBM JDK 6)"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"效果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"生效前, 可能看不到一條事務; 生效後, 可以看到每一條事務的細節."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖所示:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b7/b7cbca7e1ab6d707fd95fbfa6cdf04f4.webp","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}