Spark源代碼::Spark多線程::NettyRpcEnv.ask解讀

原創

2021-07-19 10:24

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Spark中有很多異步處理的例子，每一個地方都值得好好去審視一番，對輔助理解spark的機理以及爲自己寫出優雅的代碼都會有很大的幫助。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"NettyRpcEnv.ask解讀","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"RpcEnv作用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"NettyRpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"是","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"的在spark中的唯一一個實現。","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"是什麼呢，可以先看一下它的class頭信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"java"},"content":[{"type":"text","text":"/**\n * An RPC environment. [[RpcEndpoint]]s need to register itself with a name to [[RpcEnv]] to\n * receives messages. Then [[RpcEnv]] will process messages sent from [[RpcEndpointRef]] or remote\n * nodes, and deliver them to corresponding [[RpcEndpoint]]s. For uncaught exceptions caught by\n * [[RpcEnv]], [[RpcEnv]] will use [[RpcCallContext.sendFailure]] to send exceptions back to the\n * sender, or logging them if no such sender or `NotSerializableException`.\n *\n * [[RpcEnv]] also provides some methods to retrieve [[RpcEndpointRef]]s given name or uri.\n */\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"就是一句話，RPC的環境。在這裏，最重要的2個操作莫過於","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以去註冊","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以去異步獲取","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}],"attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"和","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"是什麼呢，在這裏不做詳細贅述，其他的文章中會詳細說明，簡單來講一下","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"簡單回顧","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"和","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}],"attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"RpcEndpoint","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"RpcEndpoint","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"衆所周知，spark內部會有","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"executor","attrs":{}}],"attrs":{}},{"type":"text","text":"，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"driver","attrs":{}}],"attrs":{}},{"type":"text","text":"等角色，他們之間的通信都採用利用Netty，在executor或者driver上並不是只啓動1個Netty的服務，針對不同的功能會有多個Netty的RPC服務開啓，利用不同的端口號進行區分。服務間通信後，通的“信”被很多種邏輯單元來處理，如","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Inbox","attrs":{}}],"attrs":{}},{"type":"text","text":"，如","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"EventLoop","attrs":{}}],"attrs":{}},{"type":"text","text":"等，這些都是工具級別的單元，而被抽象出來作爲可插拔可擴展的大的邏輯功能模塊在Spark中就叫做","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"，它是用來處理從其他","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"client","attrs":{}}],"attrs":{}},{"type":"text","text":"端發送或者","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"server","attrs":{}}],"attrs":{}},{"type":"text","text":"端返回過來的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"message","attrs":{}}],"attrs":{}},{"type":"text","text":"的模塊。","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"本身是一個trait，它可以有多種的實現","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"RpcEndpointRef","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"spark之前的網絡通信都是採用akka，改版後採用的是Netty，在akka中，如果一個兩個節點間的通信是利用目的方的actorRef來進行的通信的，即AActor 希望發送消息到 BActor，需要BActorRef來發送消息。Spark的網絡通信升級到Netty後，Endpoint就可以間接理解成原來的Actor，那麼發送消息到另一個Actor的話，也需要","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"的Ref，即","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"。這個概念乍一看有點懵，試想，從A發送消息到B，能發送的前提是A先擁有了一個B的”引用“，這在普通的Http服務中貌似很不能被理解，我想訪問某一臺機器按說只需要知道對方的IP和Port不就OK了，現在還需要對方的一個“替身”？這是什麼鬼？帶着問題我們可以持續往下看即可，這裏你只需要這樣意識即可：","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"用來訪問B machine的","attrs":{}},{"type":"codeinline","content":[{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"RpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"你理解成就是B machine的IP和Port的一個被包裝後的實例即可","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"圖解","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"和","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}],"attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"圖解一下","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A machine可以是物理機可以是虛擬機，B machine可以是和A同一臺物理機、虛擬機（端口號不同），也可以是不同的（在spark中甚至於有自己發給自己的msg，後續會講）。那麼從A發送消息到B的話，使用的是B的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"，通過它發送消息到B machine","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"【圖1】","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"要如何訪問","attrs":{}}]}]}],"attrs":{}},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/38/38d663ccc569ce30c90304b9d67a8ac4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"【圖2】","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"內部的原理","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5b/5b3f0f1209c4560e0d3a230e7d464f04.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"【圖3】","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"B machine的RpcEndpointRef的實例是啥（簡化版）","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b8/b8b41e646727257263f49f6317664c1a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Driver和Executor","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"k，顧名思義——問。可能是打個招呼，看看在不在，詢問一下，等等。這個就是NettyRpcEnv.ask的作用所在。爲了講NettyRpcEnv.ask的作用，還需要簡單的串一下一下概念和流程","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Driver線程和Executor進程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，需要明確兩個事情，在yarn環境下","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"是在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicatioMaster","attrs":{}}],"attrs":{}},{"type":"text","text":"進程中執行的一個線程","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"嚴格來說，其實這個說法也不太正確，因爲","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"其實是在用戶的class的時候，在形成","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"sparkContext","attrs":{}}],"attrs":{}},{"type":"text","text":"上下文環境的一個產物，本身執行的其實是用戶class線程，在這個線程中建立了","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"SparkEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"以及","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"等等，並且建立了","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"的Netty的Service等等，與","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"相互通信","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"則是一個個的進程，通過java命令在每一個節點上啓動的","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Yarn系列以及 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicatioMaster","attrs":{}}],"attrs":{}},{"type":"text","text":"是什麼這裏不做贅述，其他文章中會細講。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次，在這裏只需要瞭解到，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"本身是一個協調調度節點，它可以去分配任務給","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"，並且掌握着","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"的情況，分配就是把","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Task","attrs":{}},{"type":"text","text":"發送給","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"，掌握則指的是需要知道","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"的運行情況等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"【圖4】","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"圖解一下","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉個栗子，1個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"和2個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"進行交互通信，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"手握2個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Executor","attrs":{}}],"attrs":{}},{"type":"text","text":"（一個叫做E1，一個叫做E2）的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"，姑且簡稱爲E1Ref和E2Ref，通過這2個Ref發送msg到E1節點和E2節點，這2個節點本身通過自身的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"來處理msg。而E1和E2本身還要定期起的向Driver彙報自身的情況，這裏叫做heartbeat心跳，那麼反過來則是利用各自內部掌握的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverEpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"來發送","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"heartbeat","attrs":{}},{"type":"text","text":"到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"，而","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"利用其自己的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverRpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"來處理","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"heartbeat","attrs":{}},{"type":"text","text":"的msg。所有節點的上面的組建則都在自身的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NettyRpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"中，也就是","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"的實現。","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/55/55ea37fa20d2410ce9828387af481e8e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"舉例：在RpcEnv中建立一個DriverRpcEndpointRef","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"終於要說到本篇的內容了，","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"NettyRpcEnv.ask","attrs":{}},{"type":"text","text":"的解讀，需要有一個場景調用","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"NettyRpcEnv.ask","attrs":{}},{"type":"text","text":"的方法纔可以，那可以在題中所述的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"中建立一個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverRpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"這個場景中描述","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"RpcEnv中爲啥建立DriverRpcEndpointRef","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面的【圖4】介紹了一個Driver和和Executor之間通信的過程。其實，在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicationMaster","attrs":{}}],"attrs":{}},{"type":"text","text":"中構建","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"線程的時候，有一部分的通信是需要通過","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverRpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"進行的，即利用","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverRpcEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"發送msg給","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverRpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverRpcEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"做出處理並響應","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"【圖5】圖解一下","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"ApplicationMaster","attrs":{}}],"attrs":{}},{"type":"text","text":" 中啓動【","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Run","attrs":{}},{"type":"text","text":"】Driver的線程後，從Driver線程中拿到了","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NettyRpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"，","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"並且利用","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"NettyRpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"setupEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"方法【","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Get","attrs":{}},{"type":"text","text":"】到兩個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverEndpointRef","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後續通過【","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Use","attrs":{}},{"type":"text","text":"】這個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"去訪問Driver的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverEndpoint","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有一點需要說明的是，","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicationMaster","attrs":{}}],"attrs":{}},{"type":"text","text":"的節點本身也是","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"的節點，其實訪問","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverEndpoint","attrs":{}}],"attrs":{}},{"type":"text","text":"按說是可以直接訪問的（Spark源代碼中沒有這樣實現，還是爲了隔離和封裝的更好，減少耦合，今後","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"如果作爲進程執行，不再","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicationMaster","attrs":{}}],"attrs":{}},{"type":"text","text":"上運行也會修改的較爲簡單），但是這裏還是採用了Netty的Rpc訪問方式","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a7/a750a5fceb21a2b9b2a970e0fe39d969.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"源代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這部分代碼在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicationMaster.scala","attrs":{}}],"attrs":{}},{"type":"text","text":"中，關注方法","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"runDriver","attrs":{}}],"attrs":{}},{"type":"text","text":"即可","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"【圖6】圖解一下","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(I) 有一臺IP是10.1.2.5的服務器，啓動了","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicationMaster","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(II) **a過程，**在這個節點上啓動了Driver的線程，並且初始化了用戶的class，並且在10.1.2.5節點上啓動了一個Netty的serviec，IP和Port爲10.1.2.5:13200","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(III) b**過程，**在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"ApplicationMaster","attrs":{}}],"attrs":{}},{"type":"text","text":"節點上繼續調用","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"RpcEnv.setupEndpointRef","attrs":{}},{"type":"text","text":"，目的是setup一個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Driver","attrs":{}}],"attrs":{}},{"type":"text","text":"的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"到","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RpcEnv","attrs":{}}],"attrs":{}},{"type":"text","text":"中，這個setup的過程就是去10.1.2.5:13200訪問一下，如果服務通了，則構建出","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"DriverEndpointRef","attrs":{}}],"attrs":{}},{"type":"text","text":"，這個“訪問一下”即本文所述要用到的","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"NettyRpcEnv.ask","attrs":{}},{"type":"text","text":"的方法。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"可以看到調用順序爲","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(ApplicationMaster.scala) rpcEnv.setupEndpointRef ↓","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(NettyRpcEnv.scala) NettyRpcEnv.asyncSetupEndpointRefByURI ↓","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(NettyRpcEndpointRef.scala) NettyRpcEndpointRef.ask ↓","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"(NettyRpcEnv.scala) NettyRpcEnv.ask — — — — ↓ （經過多個步驟，中間部分省略，其他文章會講）","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"10.1.2.5:13200 的netty服務","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/91/917d0b73f3a02a28ce25d0f6c85f8dd5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"代碼如下","attrs":{}}]}]}],"attrs":{}},{"type":"codeblock","attrs":{"lang":"scala"},"content":[{"type":"text","text":" private def runDriver(): Unit = {\n addAmIpFilter(None)\n\n /*\n 這裏，調用startUserApplication方法來執行用戶的class，也就是我們的jar包，\n invoke我們的main方法，從而啓動了sparkContext，內部啓動一系列的scheduler以及\n backend，以及taskscheduler等等等等core的內容，其他篇章會詳細講解\n */ \n userClassThread = startUserApplication()\n\n // This a bit hacky, but we need to wait until the spark.driver.port property has\n // been set by the Thread executing the user class.\n logInfo(\"Waiting for spark context initialization...\")\n val totalWaitTime = sparkConf.get(AM_MAX_WAIT_TIME)\n try {\n\n /*\n 這裏，阻塞的等待SparkContext從Driver線程中返回回來\n */\n val sc = ThreadUtils.awaitResult(sparkContextPromise.future,\n Duration(totalWaitTime, TimeUnit.MILLISECONDS))\n if (sc != null) {\n rpcEnv = sc.env.rpcEnv\n\n val userConf = sc.getConf\n val host = userConf.get(\"spark.driver.host\")\n val port = userConf.get(\"spark.driver.port\").toInt\n registerAM(host, port, userConf, sc.ui.map(_.webUrl))\n\n /*\n **這裏，上演了好戲，通過NettyRpcEnv的setupEndpointRef方法來獲取到driverRef\n 這個裏面其實是去ask一下Driver你在嗎？是否存在這個Driver的服務，如果存在，則\n 返回OK，構建出Driver的Ref**\n */\n val driverRef = rpcEnv.setupEndpointRef(\n RpcAddress(host, port),\n YarnSchedulerBackend.ENDPOINT_NAME)\n createAllocator(driverRef, userConf)\n } else {\n // Sanity check; should never happen in normal operation, since sc should only be null\n // if the user app did not create a SparkContext.\n throw new IllegalStateException(\"User did not initialize spark context!\")\n }\n resumeDriver()\n userClassThread.join()\n } catch {\n case e: SparkException if e.getCause().isInstanceOf[TimeoutException] =>\n logError(\n s\"SparkContext did not initialize after waiting for $totalWaitTime ms. \" +\n \"Please check earlier log output for errors. Failing the application.\")\n finish(FinalApplicationStatus.FAILED,\n ApplicationMaster.EXIT_SC_NOT_INITED,\n \"Timed out waiting for SparkContext.\")\n } finally {\n resumeDriver()\n }\n }\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"解讀NettyRpcEnv.ask","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"回顧Future","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何理解Future呢，從字面意思可以很好的理解，Future即未來，也是期貨的意思。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說到期貨，就充滿了不確定性，因爲畢竟沒有發生，誰也不知道未來會怎樣。所以，定義一個Future就是定義了一個不在現在這個時空（線程）發生的（未來）的另一個（另一個線程的）事件，相比java的雞肋的Future，scala的Future可謂是非常優雅且完美，搜索我的博客可以看到針對scala的Future的詳細介紹。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"官方文章","attrs":{}},{"type":"text","text":"：","attrs":{}},{"type":"link","attrs":{"href":"https://docs.scala-lang.org/zh-cn/overviews/core/futures.html","title":"","type":null},"content":[{"type":"text","text":"https://docs.scala-lang.org/zh-cn/overviews/core/futures.html","attrs":{}}]}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏不從源代碼的角度去構建Future和Promise的認知觀念，會有其他的文章再做解釋","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"【圖7】圖解一下","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在java中定義一個線程是右側的做法，而在左側的scala中，利用Future則優雅了很多","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f1/f14ee590972a2a14c5ec970ffc9d5177.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"代碼","attrs":{}}]}]}],"attrs":{}},{"type":"codeblock","attrs":{"lang":"scala"},"content":[{"type":"text","text":" import scala.concurrent.ExecutionContext.Implicits.global\n import scala.concurrent.Future\n\n /**\n * 解讀Future的基礎\n */\n object DocFutureTest {\n\n def apply(): Unit = {\n println(\"I am DocFutureTest\")\n }\n\n def main(args: Array[String]): Unit = {\n\n val sleeping = 3000;\n val main_thread = Thread.currentThread().getName;\n \n /*\n 定義另一個線程發生的事件\n 這個事件相當於java中的如下的代碼塊：\n 從整體的間接性上看，scala的更爲優雅一些，直接一個Future可以包裹住左右需要處理的內容\n 後續如果需要進行異常處理的話還可以根據Success和Failture進行模式匹配\n public class JavaThreading {\n \n public static void main(String[] args) throws InterruptedException {\n new Thread(\n () -> System.out.println(\"這是一條發生在另一個叫做叫做\" + Thread.currentThread().getName() + \" 線程的故事\")\n ).start();\n \n System.out.println(Thread.currentThread().getName());\n Thread.sleep(3000);\n }\n }\n */\n var future_run = Future {\n Thread.sleep(1000)\n println(\"這是一條發生在另一個叫做叫做\" + Thread.currentThread().getName +\" 線程的故事\")\n }\n\n // 主線程休息3000ms\n // 如果不休息的話，main線程會先停止，導致上面的Future定義的thread還沒有被執行到就結束了\n Thread.sleep(sleeping)\n println(s\"$main_thread 線程休息 $sleeping 毫秒\")\n\n }\n\n }\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Future + callback(截取部分)","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"scala"},"content":[{"type":"text","text":" case class ExceptionError(error: String) extends Exception(error)\n\n def main(args: Array[String]): Unit = {\n\n val sleeping = 3000;\n val main_thread = Thread.currentThread().getName;\n\n // 定義另一個線程發生的事件\n var future_run = Future {\n Thread.sleep(1000)\n prntln(\"這是一條發生在另一個叫做叫做\" + Thread.currentThread().getName + \" 線程的故事\")\n // 如果需要onFailure的話則釋放此句\n // throw ExceptionError(\"error\") \n\n future_run onFailure {\n case t => println(\"exception \" + t.getMessage)\n }\n\n future_run onSuccess {\n case _ => println(\"success\")\n }\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"注意點","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定義了Future，則定義了需要執行的線程的執行體（body），那麼執行也是立刻馬上，類似於java定義了一個Thread，然後直接調用了","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"start()","attrs":{}}],"attrs":{}},{"type":"text","text":"一樣","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Future中大量運用了scala的Try[]，如果出現了異常，沒有做","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"onFailure","attrs":{}}],"attrs":{}},{"type":"text","text":"的處理，那麼可能看不到異常被拋出來，這點和java有較大區別","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"回顧Promise","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從皮毛簡單的說完了Future，那Promise又是什麼呢？其實在Future的實現中包含了Promise的實現，也就是說沒有Promise，Future是無法被運行的。從字面的理解，Promise是承諾，有了Future的未來的定義，那麼需要給出一個確切的承諾纔可以進行，否則都是空口無憑天馬行空無法兌現的大話。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"官方文章","attrs":{}},{"type":"text","text":"：","attrs":{}},{"type":"link","attrs":{"href":"https://docs.scala-lang.org/zh-cn/overviews/core/futures.html","title":"","type":null},"content":[{"type":"text","text":"https://docs.scala-lang.org/zh-cn/overviews/core/futures.html","attrs":{}}]}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說到現在，包括看完以上的Future的介紹，很多人肯定還是懵b狀態，因爲我剛開始接觸的時候也是這樣，但我喜歡的就是用最直觀的圖和想象來描述一個抽象的問題，二話不說，繼續上圖","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"【圖8】","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"圖解一下","attrs":{}},{"type":"text","text":"：","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Future","attrs":{}},{"type":"text","text":"與","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Promise","attrs":{}},{"type":"text","text":"的關係","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Future的含義","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主線人生是你的Main Thread，在spark中可能是某一個處理的Thread","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Now這個時刻，你開啓了變成star之路（become star的Thread）","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Now這個時刻，你開啓了變帥之路（become handsome的Thread）","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一旦你開啓這兩條路，只要你的Main Thread沒有結束，那麼你可以一直持續的去走完這兩個“之路”，直到","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Success","attrs":{}},{"type":"text","text":"或者","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Failure","attrs":{}},{"type":"text","text":"，這就是Future，可以理解爲，開啓了一條新的軌跡","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fb/fbc41a05c0d961669c21edda3f63842d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Promiose的含義","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當你開啓了兩條新的“之路”的時候，我可以在你兩條路的重點給你不同的承諾","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當你success的完成了Future的時候，我promise你一個結果","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當你failure的完成了Future的時候，我promise你另一個結果","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/32/3247dc7a89b69f7a1384012e6a833156.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Future與Promise的對比","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Future是一條線，包含執行過程的一條線，按照Timeline要去走下去","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":2,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Promise是一個點，一個被觸發的點，想達到這個點必須又一個Future搭出一條路徑纔可以","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面兩句話如何理解呢，你可以這麼想，人生（Main Thread）是一個數軸，你如果希望按照timeline向着右側一直前進就需要有一條連續的道路，這個“道路”就是一個Thread，也可以是Future定義出的道路，我們只能腳踏實地的通過道路走到目標終點，而不能直接跳到終點。Promise類似於一個milestone點，如果只有一個Promise，不定義出“道路”也就是不定義出一個Future（或Main Thread）的話，是無法實現這個Promise的。只定義了Promise，不去考慮直線路徑（Future），無法實現，但只定義Future，不定義Promise（其實在Future中是內置了Default的Promise的）是可以直接執行Future的。如下圖所示，開出了兩張空頭支票，沒有定義具體的路線（Future 實現方式），那麼這兩個Promise是無法兌現的。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意一點，這張圖只是畫出了定義了Promise，但是如果想對象這個Promise的話，是可以通過Promise中的方法來搭建出一個Future來執行的，與Future不同的是，Future只要定義了就可以馬上執行，Promise定義了的話，必須要顯式的觸發“搭建Future”的操作纔可以。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/77/77fcaf771d8378bc5e2ec409581448ee.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"看看Promise不執行的代碼","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏，我們定義了一個Promise，並且“承諾”在Promise對應的future結束後調用一個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"map","attrs":{}}],"attrs":{}},{"type":"text","text":"操作打印出一句話future:....","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但我們執行以下語句的時候會發現什麼都沒有執行","attrs":{}}]}]}],"attrs":{}},{"type":"codeblock","attrs":{"lang":"scala"},"content":[{"type":"text","text":" import scala.concurrent.Promise\n import scala.util.{Failure, Success}\n\n object PromiseTest {\n def main(args: Array[String]): Unit = {\n import scala.concurrent.ExecutionContext.Implicits.global\n val promise = Promise[String]\n promise.future.onComplete(v => println(\"onComplete \" + v))\n promise.future.map(str => println(\"future: \" + str + \" ==> \" + Thread.currentThread().getName))\n promise.future.failed.foreach(e => println(e + \" ==> \" + Thread.currentThread().getName))\n Thread.sleep(3000)\n }\n\n }\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"看看可以執行的代碼","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與上面代碼唯一不同的就是，加入了","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"promise.trySuccess","attrs":{}}],"attrs":{}},{"type":"text","text":"的處理","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"代碼細節在其他篇章我們具體看，這裏你可以這樣理解，加入和trySuccess，就是爲達到Promise搭建了一條Future之路，並觸發這條路開始執行（start）","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至於trySuccess，tryComplete等具體的細節講scala多線程的地方可以細說","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"promise.future.onComplete","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Future執行完畢後的callback處理，無論是","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Success","attrs":{}},{"type":"text","text":"還是","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Failure","attrs":{}},{"type":"text","text":"都可以執行這個","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"onComplete","attrs":{}}],"attrs":{}},{"type":"text","text":"處理","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"promise.future.map","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"promise.future.onComplete","attrs":{}}],"attrs":{}},{"type":"text","text":"之後對Future進行的繼續的map處理","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"promise.trySuccess","attrs":{}}],"attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":1,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"觸發整個Future執行的trigger","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"scala"},"content":[{"type":"text","text":" import scala.concurrent.Promise\n import scala.util.{Failure, Success}\n\n object PromiseTest {\n def main(args: Array[String]): Unit = {\n import scala.concurrent.ExecutionContext.Implicits.global\n val promise = Promise[String]\n promise.future.onComplete(v => println(\"onComplete \" + v))\n promise.future.map(str => println(\"future: \" + str + \" ==> \" + Thread.currentThread().getName))\n promise.future.failed.foreach(e => println(e + \" ==> \" + Thread.currentThread().getName))\n **promise.trySuccess(\"try success \" + \" --> \" + Thread.currentThread().getName)**\n Thread.sleep(3000)\n }\n\n }\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"ask的代碼","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其實，講完了上面的所有的內容後，ask的代碼感覺幾句話就可以講解完畢了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ask本身返回的是Future，本身是異步處理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"【圖9】圖解一下","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一臺10.1.1.1的client機器通過rpc訪問一臺10.1.1.2的Netty的service，當response正確返回後，在client機器中的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"TransportResponseHandler","attrs":{}}],"attrs":{}},{"type":"text","text":"中進行判斷處理，並且調用listener的onSuccess方法，這個onSuccess方法則是下面的ask代碼中定義的方法。在這個方法中本身又去執行了promise的tryComplete，從而觸發了promise的future之路執行","attrs":{}}]}]}],"attrs":{}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fd/fda0cbf71394fe8100dd8dd5adbc31d5.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}],"attrs":{}},{"type":"codeblock","attrs":{"lang":"scala"},"content":[{"type":"text","text":"private[netty] def ask[T: ClassTag](message: RequestMessage, timeout: RpcTimeout): Future[T] = {\n // 定義了一個Any的promise\n val promise = Promise[Any]()\n val remoteAddr = message.receiver.address\n\n def onFailure(e: Throwable): Unit = {\n if (!promise.tryFailure(e)) {\n e match {\n case e : RpcEnvStoppedException => logDebug (s\"Ignored failure: $e\")\n case _ => logWarning(s\"Ignored failure: $e\")\n }\n }\n }\n\n /* \n 這裏聲明的onSuccess會被填充到RpcResponseCallback的onSuccess中，這個\n RpcResponseCallback就是上面【圖9】中的listener，當我們從Server端獲取到response後\n 注意，獲取的不是RpcFailure類型的response，則都會進入到【圖9】的\n else if (message instanceof RpcResponse) { 分支中\n\n */\n def onSuccess(reply: Any): Unit = reply match {\n case RpcFailure(e) => onFailure(e)\n case rpcReply =>\n /*\n 當返回的response是OK的沒有問題後，onSuccess被callback，這裏promise的trySuccess也\n 進行call操作，這裏就是上面所說的，爲了一個promise鋪設了一條future，從而可以執行\n 這個Future的線程了\n */\n if (!promise.trySuccess(rpcReply)) {\n logWarning(s\"Ignored message: $reply\")\n }\n }\n\n try {\n if (remoteAddr == address) {\n val p = Promise[Any]()\n p.future.onComplete {\n case Success(response) => onSuccess(response)\n case Failure(e) => onFailure(e)\n }(ThreadUtils.sameThread)\n dispatcher.postLocalMessage(message, p)\n } else {\n val rpcMessage = RpcOutboxMessage(message.serialize(this),\n onFailure,\n (client, response) => **onSuccess**(deserialize[Any](client, response)))\n postToOutbox(message.receiver, rpcMessage)\n /*\n 如果是callback了Failure，則這裏會被執行\n */\n promise.future.failed.foreach {\n case _: TimeoutException => rpcMessage.onTimeout()\n case _ =>\n }(ThreadUtils.sameThread)\n }\n\n val timeoutCancelable = timeoutScheduler.schedule(new Runnable {\n override def run(): Unit = {\n onFailure(new TimeoutException(s\"Cannot receive any reply from ${remoteAddr} \" +\n s\"in ${timeout.duration}\"))\n }\n }, timeout.duration.toNanos, TimeUnit.NANOSECONDS)\n\n /*\n 當promise的future執行後，會調用這裏的onComplete方法\n */\n promise.future.onComplete { v =>\n timeoutCancelable.cancel(true)\n }(ThreadUtils.sameThread)\n } catch {\n case NonFatal(e) =>\n onFailure(e)\n }\n\n /*\n 利用RpcTimeout中的addMessageIfTimeout的偏函數再去模式匹配一下產生的Throwable內容\n 如果是RpcTimeoutException 則直接throw這個ex\n 如果是TimeoutException 則包裝成RpcTimeoutException後再throw出去\n */\n promise.future.mapTo[T].recover(timeout.addMessageIfTimeout)(ThreadUtils.sameThread)\n }\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本篇用小篇幅講解了一下*o.a.s.rpc.netty.NettyRpcEnv.ask()*的方法，簡單描述了一個spark的異步處理的小case，這個小case需要不少的先驗知識點，可能突然間看到這裏有點懵，學習需要融會貫通一點點的來積累纔可以，如果不明白可以慢慢積累其他模塊的知識再來這裏看流水賬會更有收穫。","attrs":{}}]}]}