轉載：https://blog.cloudera.com/hadoop-delegation-tokens-explained/
很好的文章，但是要翻牆轉載給國內的夥伴，有問題請聯繫刪除
第一部分谷歌翻譯版；下邊有英語版
Apache Hadoop的安全性是在2009年左右設計和實施的，此後一直保持穩定。但是，由於缺少有關此領域的文檔，因此出現問題時很難理解或調試。設計了委託令牌，並將其作爲身份驗證方法在Hadoop生態系統中廣泛使用。這篇博客文章介紹了Hadoop分佈式文件系統（HDFS）和Hadoop密鑰管理服務器（KMS）上下文中的Hadoop委託令牌的概念，並提供了一些基本代碼和故障排除示例。值得注意的是，Hadoop生態系統中還有許多其他服務也使用委託令牌，但爲簡潔起見，我們僅討論HDFS和KMS。
此博客假設讀者瞭解的基本概念驗證，Kerberos的，爲了理解認證流程的目的; 以及HDFS體系結構和HDFS透明加密，以瞭解什麼是HDFS和KMS。對於對HDFS透明加密不感興趣的讀者，可以忽略此博客文章中的KMS部分。可以在此處找到有關Hadoop中的一般授權和身份驗證的先前博客文章。
Hadoop安全性快速入門
Hadoop最初是在沒有真實身份驗證的情況下實施的，這意味着存儲在Hadoop中的數據很容易遭到破壞。此安全功能後來於2010年通過HADOOP-4487添加，其主要目的有以下兩個：
- 防止未經授權訪問HDFS中存儲的數據
- 在實現目標1的同時不增加大量成本。
爲了實現第一個目標，我們需要確保
- 任何訪問羣集的客戶端都經過身份驗證，以確保它們是他們聲稱的身份。
- 集羣中的所有服務器都經過身份驗證，可以成爲集羣的一部分。
爲了這個目標，選擇Kerberos作爲基礎身份驗證服務。添加了其他機制，例如委託令牌，塊訪問令牌，信任等，以補充Kerberos。特別是，引入了代幣令牌以實現第二個目標（有關方法，請參閱下一節）。以下是簡化圖，說明了在HDFS上下文中使用Kerberos和委派令牌的位置（其他服務類似）：
圖1：HDFS身份驗證機制的簡化圖
在上面的簡單HDFS示例中，有幾種身份驗證機制在起作用：
- 最終用戶（喬）可以使用Kerberos與HDFS NameNode對話
- 最終用戶（joe）提交的分佈式任務可以使用joe的委派令牌訪問HDFS NameNode。這將是本博客文章其餘部分的重點
- HDFS數據節點使用Kerberos與HDFS NameNode對話
- 最終用戶和分佈式任務可以使用“塊訪問令牌”訪問HDFS DataNode。
我們將在本博文結尾的“ 其他使用令牌的方式 ”部分中簡要介紹上述機制。要了解Hadoop的安全設計的詳細信息，請參閱設計文檔在HADOOP-4487和Hadoop的安全架構演示。
什麼是授權令牌？
從理論上講，可以完全使用Kerberos進行身份驗證，但是在像Hadoop這樣的分佈式系統中使用時，它就有其自身的問題。想象一下，對於每個MapReduce作業，如果所有工作任務都必須使用委派的TGT（票證授予票證）通過Kerberos進行身份驗證，則Kerberos密鑰分發中心（KDC）將很快成爲瓶頸。下圖中的紅線說明了這個問題：一個工作可能有成千上萬的節點到節點通信，導致相同數量的KDC流量。實際上，它會在非常大的集羣中無意中對KDC 進行分佈式拒絕服務攻擊。
圖2：顯示Hadoop中身份驗證擴展問題的簡化圖
因此，引入了委託令牌作爲一種輕量級身份驗證方法，以補充Kerberos身份驗證。Kerberos是一種三方協議。相反，委託令牌身份驗證是一種兩方身份驗證協議。
授權令牌的工作方式是：
- 客戶端最初通過Kerberos向每臺服務器進行身份驗證，然後從該服務器獲取委託令牌。
- 客戶端使用委派令牌來與服務器進行後續身份驗證，而不是使用Kerberos。
客戶端可以並且經常確實將委託令牌傳遞給其他服務（例如YARN），以便其他服務（例如映射器和化簡器）可以作爲客戶端進行身份驗證並代表客戶端運行作業。換句話說，客戶端可以將憑據“委派”給那些服務。授權令牌有一個到期時間，需要定期更新以保持其有效性。但是，它們不能無限期地更新–使用壽命最長。在過期之前，也可以取消委託令牌。
委託令牌消除了分發Kerberos TGT或密鑰表的需求，如果這些密鑰或密鑰表受到破壞，將授予對所有服務的訪問權限。另一方面，委派令牌嚴格與其相關的服務綁定在一起，並最終到期，如果暴露，則造成的損害較小。此外，委派令牌使憑證更新更加輕便。這是因爲更新的設計方式使得更新過程僅涉及更新程序和服務。令牌本身保持不變，因此不必更新已經使用令牌的所有參與者。
出於可用性的原因，委派令牌由服務器保留。HDFS NameNode將委託令牌保留到其元數據（也稱爲fsimage和編輯日誌）。KMS以ZNodes的形式將委派令牌保持在Apache ZooKeeper中。即使服務器重新啓動或故障轉移，這也可使委託令牌可用。
服務器和客戶端在處理委託令牌方面有不同的職責。以下兩個部分提供了一些詳細信息。
服務器端的委託令牌
該服務器（即圖2中的HDFS NN和KMS）負責：
- 發行委託令牌，並存儲它們以進行驗證。
- 根據請求續訂委派令牌。
- 在客戶端取消委派令牌或它們到期時刪除委派令牌。
- 通過對照存儲的委託令牌驗證提供的委託令牌來驗證客戶端。
Hadoop中的委託令牌是根據HMAC機制生成和驗證的。委託令牌中有兩部分信息：公共部分和私有部分。委託令牌以哈希圖的形式存儲在服務器端，公共信息作爲鍵，私有信息作爲值。
公共信息以標識符對象的形式用於令牌標識。它包括：
| 類 | 令牌的種類（HDFS_DELEGATION_TOKEN或kms-dt）。令牌還包含種類，與標識符的種類匹配。 |
| 所有者 | 擁有令牌的用戶。 |
| 續訂 | 可以續簽令牌的用戶。 |
| 真實用戶 | 僅在所有者被假冒時相關。如果令牌是由模擬用戶創建的，則這將標識模擬用戶。 例如，當oozie冒充用戶joe時，所有者將爲joe，而實際用戶將爲oozie。 |
| 發行日期 | 令牌發行的時代時間。 |
| 最長日期 | 令牌可以更新到的時期。 |
| 序列 號 | UUID以標識令牌。 |
| 主密鑰ID | 用於創建和驗證令牌的主密鑰的ID。 |
表1：令牌標識符（委託令牌的公共部分）
私有信息由AbstractDelegationTokenSecretManager中的類DealerTokenInformation表示，它對安全性至關重要，並且包含以下字段：
| 更新日期 | 令牌預計將更新的時期。 如果小於當前時間，則表示令牌已過期。 |
| 密碼 | 使用主密鑰作爲HMAC密鑰計算爲令牌標識符的HMAC的密碼。 它用於驗證客戶端提供給服務器的委派令牌。 |
| 跟蹤號碼 | 跟蹤標識符，可用於在多個客戶端會話之間關聯令牌的用法。 它被計算爲令牌標識符的MD5。 |
表2：委託令牌信息（委託令牌的私有部分）
請注意表1中的主密鑰ID，這是服務器上存在的主密鑰的ID。主密鑰用於生成每個委派令牌。它以配置的時間間隔滾動，並且永不離開服務器。
服務器還具有指定更新間隔的配置（默認爲24小時），這也是委託令牌的到期時間。過期的委託令牌不能用於身份驗證，並且服務器具有後臺線程，可以從服務器的令牌存儲中刪除過期的委託令牌。
在過期之前，只有授權令牌的續訂者才能續簽。成功的續訂將授權令牌的有效期延長了另一個續訂間隔，直到達到其最大生命週期（默認爲7天）。
附錄A中的表提供了HDFS和KMS的配置名稱及其默認值。
客戶端的委託令牌
客戶負責
- 從服務器請求新的委派令牌。請求令牌時可以指定續訂。
- 續訂委派令牌（如果客戶將自己指定爲“續訂者”），或要求另一方（指定的“續訂者”）續訂委派令牌。
- 請求服務器取消委派令牌。
- 出示委派令牌以與服務器進行身份驗證。
客戶端可見的Token類在此處定義。下表描述了令牌中包含的內容。
| 識別碼 | 與服務器端的公共信息部分匹配的令牌標識符。 |
| 密碼 | 與服務器端密碼匹配的密碼。 |
| 類 | 令牌的種類（例如HDFS_DELEGATION_TOKEN或kms-dt），它與標識符的種類匹配。 |
| 服務 | 服務的名稱（例如，對於HDFS爲ha-hdfs：<nameservice-name>，對於KMS爲<ip_address>：<port>）。 |
| 更新 | 可以續簽令牌（例如紗線）的用戶。 |
表3：客戶端的委託令牌
下一部分將說明使用委派令牌進行身份驗證的方式。
以下是作業提交時的示例日誌摘錄。INFO日誌打印有來自所有服務的令牌。在下面的示例中，我們看到一個HDFS委託令牌和一個KMS委託令牌。
$ hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-examples-2.6.0-cdh5.13.0.jar pi 2 3 Number of Maps = 2 Samples per Map = 3 Wrote input for Map #0 Wrote input for Map #1 Starting Job 17/10/22 20:50:03 INFO client.RMProxy: Connecting to ResourceManager at example.cloudera.com/172.31.113.88:8032 17/10/22 20:50:03 INFO hdfs.DFSClient: Created token for xiao: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1508730603423, maxDate=1509335403423, sequenceNumber=4, masterKeyId=39 on ha-hdfs:ns1 17/10/22 20:50:03 INFO security.TokenCache: Got dt for hdfs://ns1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: (token for xiao: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1508730603423, maxDate=1509335403423, sequenceNumber=4, masterKeyId=39) 17/10/22 20:50:03 INFO security.TokenCache: Got dt for hdfs://ns1; Kind: kms-dt, Service: 172.31.113.88:16000, Ident: (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730603474, maxDate=1509335403474, sequenceNumber=7, masterKeyId=69) … |
對於想要編寫Java代碼進行身份驗證的讀者，附錄B提供了示例代碼。
示例：委託令牌的生命週期
現在我們瞭解了委託令牌是什麼，讓我們看一下它在實踐中的用法。下圖顯示了用於運行典型應用程序的身份驗證示例流程，其中該作業是通過YARN提交的，然後被分發到集羣中的多個工作程序節點以執行。
圖3：在典型作業中如何使用委託令牌進行身份驗證
爲簡便起見，省略了Kerberos身份驗證的步驟以及有關任務分發的詳細信息。圖中一般有5個步驟：
- 客戶端要在集羣中運行作業。它從HDFS NameNode獲取一個HDFS委託令牌，並從KMS獲得一個KMS委託令牌。
- 客戶端將作業提交給YARN資源管理器（RM），並傳遞剛獲取的委派令牌以及ApplicationSubmissionContext。
- YARN RM通過立即續簽來驗證委託令牌是否有效。然後，它將啓動作業，該作業（與委託令牌一起）分發到集羣中的工作節點。
- 每個工作節點在訪問HDFS數據時都使用HDFS委派令牌向HDFS進行身份驗證，並在解密加密區域中的HDFS文件時使用KMS委派令牌向KMS進行身份驗證。
- 作業完成後，RM取消該作業的委派令牌。
注意：上圖中未繪製的步驟是，RM還將跟蹤每個委託令牌的到期時間，並在到期時間的90％時更新委託令牌。請注意，在RM中會單獨跟蹤委派令牌，而不是基於令牌種類。這樣，具有不同更新間隔的令牌都可以正確更新。令牌更新類是使用Java ServiceLoader實現的，因此RM不必知道令牌的種類。對於非常感興趣的讀者，相關代碼在RM的PrincipledTokenRenewer類中。
這是什麼InvalidToken錯誤？
現在我們知道什麼是委託令牌，以及在執行典型任務時如何使用它們。但是不要在這裏停下來！讓我們從應用程序日誌中查看一些典型的與委託令牌相關的錯誤消息，並弄清它們的含義。
令牌已過期
有時，應用程序會因AuthenticationException而失敗，並在其中封裝了InvalidToken異常。異常消息指示“令牌已過期”。猜猜爲什麼會發生這種情況？
|
… 17/10/22 20:50:12 INFO mapreduce.Job: Job job_1508730362330_0002 failed with state FAILED due to: Application application_1508730362330_0002 failed 2 times due to AM Container for appattempt_1508730362330_0002_000002 exited with exitCode: -1000 |
在緩存中找不到令牌
有時，應用程序會因AuthenticationException而失敗，並在其中封裝了InvalidToken異常。異常消息指示“在緩存中找不到令牌”。猜猜爲什麼會發生這種情況，與“令牌已過期”錯誤有什麼區別？
| …。 17/10/22 20:55:47 INFO mapreduce.Job: Job job_1508730362330_0003 failed with state FAILED due to: Application application_1508730362330_0003 failed 2 times due to AM Container for appattempt_1508730362330_0003_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:https://example.cloudera.com:8090/proxy/application_1508730362330_0003/Then, click on links to logs of each attempt. Diagnostics: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730937041, maxDate=1509335737041, sequenceNumber=8, masterKeyId=73) can’t be found in cache java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730937041, maxDate=1509335737041, sequenceNumber=8, masterKeyId=73) can’t be found in cache at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:294) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:528) at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1448) … at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:784) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:367) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357) … at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730937041, maxDate=1509335737041, sequenceNumber=8, masterKeyId=73) can’t be found in cache … at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:535) … … |
說明：
上面的2個錯誤具有相同的原因：用於身份驗證的令牌已過期，不能再用於身份驗證。
第一條消息能夠明確告知令牌已過期，因爲該令牌仍存儲在服務器中。因此，當服務器驗證令牌時，對到期時間的驗證將失敗，並引發“令牌已過期”異常。
現在您可以猜測何時會發生第二個錯誤嗎？請記住，在“服務器端的委託令牌”部分中，我們解釋了服務器具有後臺線程來刪除過期的令牌。因此，如果在服務器的後臺線程刪除令牌後將令牌用於身份驗證，則服務器上的驗證邏輯將找不到該令牌。這導致拋出異常，說“找不到令牌”。
下圖顯示了這些事件的順序。
圖4：委託令牌的生命週期
請注意，當令牌被明確取消後，它將立即從商店中刪除。因此，對於已取消的令牌，錯誤將始終是“找不到令牌”。
爲了進一步調試這些錯誤，有必要在客戶端日誌和服務器日誌中添加特定令牌序列號（在上述示例中爲“ sequenceNumber = 7”或“ sequenceNumber = 8”）。您應該能夠在服務器日誌中看到與令牌創建，續訂（如果有），取消（如果有）有關的事件。
長期運行的應用程序
至此，您已經瞭解了有關Hadoop授權令牌的所有基礎知識。但除此之外，還有一個缺失的環節：我們知道，授權令牌無法在其最大生存期之後進行更新，那麼對於需要運行超過最大生存期的應用程序，會發生什麼呢？
壞消息是，Hadoop沒有爲所有應用程序提供統一的方法來執行此操作，因此不存在任何應用程序都可以打開以使其“正常運行”的魔術配置。但這仍然是可能的。對於Spark提交的應用程序來說，好消息是，Spark已經實現了這些神奇的參數。Spark獲取委託令牌並將其用於身份驗證，類似於我們在前面的部分中所述。但是，Spark不會續簽令牌，而只是在即將過期時獲取新令牌。這使應用程序可以永遠運行。相關代碼在這裏。請注意，這需要爲您的Spark應用程序提供Kerberos密鑰表。
但是，如果您要實現一個長期運行的服務應用程序，並希望該應用程序顯式處理令牌，該怎麼辦？這將涉及兩個部分：更新令牌，直到達到最大生存期；在最大使用壽命後處理令牌替換。
請注意，這不是通常的做法，只有在實現服務時才建議這樣做。
實施令牌續訂
讓我們首先研究應該如何進行令牌更新。最好的方法是研究YARN RM的PrincipledTokenRenewer代碼。
該類的一些關鍵點是：
- 它是管理所有令牌的單一類。在內部，它有一個續訂令牌的線程池，另一個有取消令牌的線程。更新在到期時間的90％發生。取消會有一個小的延遲（30秒），以防止比賽。
- 每個令牌的到期時間是分開管理的。通過調用令牌的renew（）API以編程方式檢索令牌的到期時間。該API的返回值是到期時間。
dttr.expirationDate =
UserGroupInformation.getLoginUser().doAs(
new PrivilegedExceptionAction<Long>() {
@Override
public Long run() throws Exception {
return dttr.token.renew(dttr.conf);
}
});
這是YARN RM收到令牌後立即更新令牌的另一個原因-知道令牌何時到期。
3.通過解碼令牌的標識符並調用其getMaxDate（）API來檢索每個令牌的最大生存期 。標識符中的其他字段也可以通過調用類似的API獲得。
if (token.getKind().equals(HDFS_DELEGATION_KIND)) {
try {
AbstractDelegationTokenIdentifier identifier =
(AbstractDelegationTokenIdentifier) token.decodeIdentifier();
maxDate = identifier.getMaxDate();
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
4.由於2和3，無需讀取配置即可確定更新間隔和最大壽命。服務器的配置可能會不時更改。客戶不應該依賴它，也沒有辦法知道它。
Max Lifetime之後處理令牌
令牌續簽將僅續簽令牌，直至其最大使用壽命。最長壽命後，該作業將失敗。如果您的應用程序是長期運行的，則應考慮使用YARN文檔中描述的有關長期服務的機制，或者在自己的委託令牌更新類中添加邏輯，以便在現有委託令牌即將達到最大值時檢索新的委託令牌。一生。
其他使用令牌的方式
恭喜你！現在，您已經閱讀完有關委託令牌的概念和詳細信息。本博客文章中未涉及一些相關方面。以下是它們的簡要介紹。
其他服務中的委託令牌：Apache Oozie，Apache Hive和Apache Hadoop的YARN RM等服務也利用委託令牌進行身份驗證。
塊訪問令牌：HDFS客戶端通過首先聯繫NameNode來獲取文件的塊位置，然後直接在DataNode上訪問這些塊來訪問文件。文件權限檢查在NameNode中進行。但是，對於隨後在DataNodes上進行的數據塊訪問，也需要進行授權檢查。塊訪問令牌用於此目的。它們由HDFS NameNode發佈給客戶端，然後由客戶端傳遞給DataNode。阻止訪問令牌的壽命很短（默認情況下爲10小時），無法更新。如果“阻止訪問令牌”到期，則客戶端必須請求一個新令牌。
身份驗證令牌：我們僅介紹了授權令牌。Hadoop還具有身份驗證令牌的概念，該令牌被設計爲更便宜且可擴展的身份驗證機制。它就像服務器和客戶端之間的cookie。身份驗證令牌是由服務器授予的，不能被更新或用於模擬其他人。並且與委託令牌不同，不需要由服務器單獨存儲。您不需要針對身份驗證令牌顯式編碼。
結論
委託令牌在Hadoop生態系統中扮演重要角色。現在，您應該瞭解委託令牌的用途，如何使用它們以及爲什麼以這種方式使用它們。在編寫和調試應用程序時，此知識必不可少。
附錄
附錄A.服務器端的配置。
下表是與委託令牌相關的配置表。請參閱服務器端的委託令牌以獲取這些屬性的說明。
| 屬性 | HDFS中的 配置名稱 | HDFS中的 默認 值 | KMS中的 配置名稱 | KMS中的 默認 值 |
| 更新 間隔 | dfs.namenode.delegation.token.renew- 間隔 |
86400000 （1天） |
hadoop.kms.authentication.delegation-token.renew- interval.sec |
86400 （1天） |
| 最長壽命 | dfs.namenode.delegation.token.max- 生存期 |
604800000 （7天） |
hadoop.kms.authentication.delegation-token.max- lifetime.sec |
604800 （7天） |
| 後臺 刪除 過期令牌的 間隔 |
不可配置 | 3600000 （1小時） |
hadoop.kms.authentication.delegation-token.removal -scan-interval.sec |
3600 （1小時） |
| 主密鑰滾動間隔 | dfs.namenode.delegation.key.update- 間隔 |
86400000 （1天） |
hadoop.kms.authentication.delegation-token.update- interval.sec |
86400 （1天） |
表4：HDFS和KMS的配置屬性和默認值
附錄B.使用委託令牌進行身份驗證的示例代碼。
在查看下面的代碼之前，需要理解的一個概念是UserGroupInformation（UGI）類。UGI是Hadoop的公共API，用於針對身份驗證進行編碼。它在下面的代碼示例中使用，並且還早些時候出現在某些異常堆棧跟蹤中。
GetFileStatus用作使用UGI訪問HDFS的示例。有關詳細信息，請參見FileSystem類javadoc。
UserGroupInformation tokenUGI = UserGroupInformation.createRemoteUser("user_name");
UserGroupInformation kerberosUGI = UserGroupInformation.loginUserFromKeytabAndReturnUGI("principal_in_keytab", "path_to_keytab_file");
Credentials creds = new Credentials();
kerberosUGI.doAs((PrivilegedExceptionAction<Void>) () -> {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.getFileStatus(filePath); // ← kerberosUGI can authenticate via Kerberos
// get delegation tokens, add them to the provided credentials. set renewer to ‘yarn’
Token<?>[] newTokens = fs.addDelegationTokens("yarn", creds);
// Add all the credentials to the UGI object
tokenUGI.addCredentials(creds);
// Alternatively, you can add the tokens to UGI one by one.
// This is functionally the same as the above, but you have the option to log each token acquired.
for (Token<?> token : newTokens) {
tokenUGI.addToken(token);
}
return null;
});
請注意，使用Kerberos身份驗證調用addDelegationTokens RPC調用。否則，將導致拋出IOException，文字爲“只能使用kerberos或Web身份驗證才能發佈委託令牌”。
現在，我們可以使用獲取的委託令牌來訪問HDFS。
tokenUGI.doAs((PrivilegedExceptionAction<Void>) () -> {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.getFileStatus(filePath); // ← tokenUGI can authenticate via Delegation Token
return null;
});
---------------------------------------------------------下面是英文版------------------------------------------------------------------
Apache Hadoop’s security was designed and implemented around 2009, and has been stabilizing since then. However, due to a lack of documentation around this area, it’s hard to understand or debug when problems arise. Delegation tokens were designed and are widely used in the Hadoop ecosystem as an authentication method. This blog post introduces the concept of Hadoop Delegation Tokens in the context of Hadoop Distributed File System (HDFS) and Hadoop Key Management Server (KMS), and provides some basic code and troubleshooting examples. It is noteworthy that there are a lot of other services in the Hadoop ecosystem that also utilize delegation tokens, but for brevity we will only discuss about HDFS and KMS.
This blog assumes the readers understand the basic concept of authentication, Kerberos, for the purpose of understanding the authentication flow; as well as HDFS Architecture and HDFS Transparent Encryption, for the purpose of understanding what HDFS and KMS are. For readers who are not interested in HDFS transparent encryption, the KMS portions in this blog post can be ignored. A previous blog post about general Authorization and Authentication in Hadoop can be found here.
Quick Introduction to Hadoop Security
Hadoop was initially implemented without real authentication, which means data stored in Hadoop could be easily compromised. The security feature was added on later via HADOOP-4487 in 2010 with the following two fundamental goals:
- Preventing the data stored in HDFS from unauthorized access
- Not adding significant cost while achieving goal #1.
In order to achieve the first goal, we need to ensure
- Any clients accessing the cluster are authenticated to ensure they are who they claimed to be.
- Any servers of the cluster are authenticated to be part of the cluster.
For this goal, Kerberos was chosen as the underlying authentication service. Other mechanisms such as Delegation Token, Block Access Token, Trust etc. are added to complement Kerberos. Especially, Delegation Token is introduced to achieve the second goal (see next section for how). Below is a simplified diagram that illustrates where Kerberos and Delegation Tokens are used in the context of HDFS (other services are similar):
Figure 1: Simplified Diagram of HDFS Authentication Mechanisms
In a simple HDFS example above, there are several authentication mechanisms in play:
- The end user (joe) can talk to the HDFS NameNode using Kerberos
- The distributed tasks the end user (joe) submits can access the HDFS NameNode using joe’s Delegation Tokens. This will be the focus for the rest of this blog post
- The HDFS DataNodes talk to the HDFS NameNode using Kerberos
- The end user and the distributed tasks can access HDFS DataNodes using Block Access Tokens.
We will give a brief introduction of some of the above mechanisms in the Other Ways Tokens Are Used section at the end of this blog post. To read about more details of Hadoop security design, please refer to the design doc in HADOOP-4487 and the Hadoop Security Architecture Presentation.
What is a Delegation Token?
While it’s theoretically possible to solely use Kerberos for authentication, it has its own problem when being used in a distributed system like Hadoop. Imagine for each MapReduce job, if all the worker tasks have to authenticate via Kerberos using a delegated TGT (Ticket Granting Ticket), the Kerberos Key Distribution Center (KDC) would quickly become the bottleneck. The red lines in the graph below demonstrates the issue: there could be thousands of node-to-node communications for a job, resulting in the same magnitude of KDC traffic. In fact, it would inadvertently be performing a distributed denial of service attack on the KDC in very large clusters.
Figure 2: Simplified Diagram Showing The Authentication Scaling Problem in Hadoop
Thus Delegation Tokens were introduced as a lightweight authentication method to complement Kerberos authentication. Kerberos is a three-party protocol; in contrast, Delegation Token authentication is a two-party authentication protocol.
The way Delegation Tokens works is:
- The client initially authenticates with each server via Kerberos, and obtains a Delegation Token from that server.
- The client uses the Delegation Tokens for subsequent authentications with the servers instead of using Kerberos.
The client can and very often does pass Delegation Tokens to other services (such as YARN) so that the other services (such as the mappers and reducers) can authenticate as, and run jobs on behalf of, the client. In other words, the client can ‘delegate’ credentials to those services. Delegation Tokens have an expiration time and require periodic renewals to keep their validity. However, they can’t be renewed indefinitely – there is a max lifetime. Before it expires, the Delegation Token can be cancelled too.
Delegation Tokens eliminate the need to distribute a Kerberos TGT or keytab, which, if compromised, would grant access to all services. On the other hand, a Delegation Token is strictly tied to its associated service and eventually expires, causing less damage if exposed. Moreover, Delegation Tokens make credential renewal more lightweight. This is because the renewal is designed in such a way that only the renewer and the service are involved in the renewal process. The token itself remains the same, so all parties already using the token do not have to be updated.
For availability reasons, the Delegation Tokens are persisted by the servers. HDFS NameNode persists the Delegation Tokens to its metadata (aka. fsimage and edit logs). KMS persists the Delegation Tokens into Apache ZooKeeper, in the form of ZNodes. This allows the Delegation Tokens to be usable even if the servers restart or failover.
The server and the client have different responsibilities for handling Delegation Tokens. The two sections below provide some details.
Delegation Tokens at the Server-side
The server (that is, HDFS NN and KMS in Figure 2.) is responsible for:
- Issuing Delegation Tokens, and storing them for verification.
- Renewing Delegation Tokens upon request.
- Removing the Delegation Tokens either when they are canceled by the client, or when they expire.
- Authenticating clients by verifying the provided Delegation Tokens against the stored Delegation Tokens.
Delegation Tokens in Hadoop are generated and verified following the HMAC mechanism. There are two parts of information in a Delegation Token: a public part and a private part. Delegation Tokens are stored at the server side as a hashmap with the public information as the key and the private information as the value.
The public information is used for token identification in the form of an identifier object. It consists of:
| Kind | The kind of token (HDFS_DELEGATION_TOKEN, or kms-dt). The token also contains the kind, matching the identifier’s kind. |
| Owner | The user who owns the token. |
| Renewer | The user who can renew the token. |
| Real User | Only relevant if the owner is impersonated. If the token is created by an impersonating user, this will identify the impersonating user. For example, when oozie impersonates user joe, Owner will be joe and Real User will be oozie. |
| Issue Date | Epoch time when the token was issued. |
| Max Date | Epoch time when the token can be renewed until. |
| Sequence Number | UUID to identify the token. |
| Master Key ID | ID of the master key used to create and verify the token. |
Table 1: Token Identifier (public part of a Delegation Token)
The private information is represented by class DelegationTokenInformation in AbstractDelegationTokenSecretManager, it is critical for security and contains the following fields:
| renewDate | Epoch time when the token is expected to be renewed. If it’s smaller than the current time, it means the token has expired. |
| password | The password calculated as HMAC of Token Identifier using master key as the HMAC key. It’s used to validate the Delegation Token provided by the client to the server. |
| trackingId | A tracking identifier that can be used to associate usages of a token across multiple client sessions. It is computed as the MD5 of the token identifier. |
Table 2: Delegation Token Information (private part of a Delegation Token)
Notice the master key ID in Table 1, which is the ID of the master key living on the server. The master key is used to generate every Delegation Token. It is rolled at a configured interval, and never leaves the server.
The server also has a configuration to specify a renew-interval (default is 24 hours), which is also the expiration time of the Delegation Token. Expired Delegation Tokens cannot be used to authenticate, and the server has a background thread to remove expired Delegation Tokens from the server’s token store.
Only the renewer of the Delegation Token can renew it, before it expires. A successful renewal extends the Delegation Token’s expiration time for another renew-interval, until it reaches its max lifetime (default is 7 days).
The table in Appendix A gives the configuration names and their default values for HDFS and KMS.
Delegation tokens at the client-side
The client is responsible for
- Requesting new Delegation Tokens from the server. A renewer can be specified when requesting the token.
- Renewing Delegation Tokens (if the client specifies itself as the ‘renewer’), or ask another party (the specified ‘renewer’) to renew Delegation Tokens.
- Requesting the server to cancel Delegation Tokens.
- Present a Delegation Token to authenticate with the server.
The client-side-visible Token class is defined here. The following table describes what’s contained in the Token.
| identifier | The token identifier matching the public information part at the server side. |
| password | The password matching the password at the server side. |
| kind | The kind of token (e.g. HDFS_DELEGATION_TOKEN, or kms-dt), it matches the identifier’s kind. |
| service | The name of the service (e.g. ha-hdfs:<nameservice-name> for HDFS, <ip_address>:<port> for KMS). |
| renewer | The user who can renew the token (e.g. yarn). |
Table 3: Delegation Token at Client Side
How authentication with Delegation Tokens works is explained in the next section.
Below is an example log excerpt at job submission time. The INFO logs are printed with tokens from all services. In the example below, we see an HDFS Delegation Token, and a KMS Delegation Token.
| $ hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-examples-2.6.0-cdh5.13.0.jar pi 2 3 Number of Maps = 2 Samples per Map = 3 Wrote input for Map #0 Wrote input for Map #1 Starting Job 17/10/22 20:50:03 INFO client.RMProxy: Connecting to ResourceManager at example.cloudera.com/172.31.113.88:8032 17/10/22 20:50:03 INFO hdfs.DFSClient: Created token for xiao: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1508730603423, maxDate=1509335403423, sequenceNumber=4, masterKeyId=39 on ha-hdfs:ns1 17/10/22 20:50:03 INFO security.TokenCache: Got dt for hdfs://ns1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: (token for xiao: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1508730603423, maxDate=1509335403423, sequenceNumber=4, masterKeyId=39) 17/10/22 20:50:03 INFO security.TokenCache: Got dt for hdfs://ns1; Kind: kms-dt, Service: 172.31.113.88:16000, Ident: (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730603474, maxDate=1509335403474, sequenceNumber=7, masterKeyId=69) … |
For readers who want to write Java code to authenticate, a sample code is provided at Appendix B.
Example: Delegation Tokens’ Lifecycle
Now that we understand what a Delegation Token is, let’s take a look at how it’s used in practice. The graph below shows an example flow of authentications for running a typical application, where the job is submitted through YARN, then is distributed to multiple worker nodes in the cluster to execute.
Figure 3: How Delegation Token is Used for Authentication In A Typical Job
For brevity, the steps of Kerberos authentication and the details about the task distribution are omitted. There are 5 steps in general in the graph:
- The client wants to run a job in the cluster. It gets an HDFS Delegation Token from HDFS NameNode, and a KMS Delegation Token from the KMS.
- The client submits the job to the YARN Resource Manager (RM), passing the Delegation Tokens it just acquired, as well as the ApplicationSubmissionContext.
- YARN RM verifies the Delegation Tokens are valid by immediately renewing them. Then it launches the job, which is distributed (together with the Delegation Tokens) to the worker nodes in the cluster.
- Each worker node authenticates with HDFS using the HDFS Delegation Token as they access HDFS data, and authenticates with KMS using the KMS Delegation Token as they decrypt the HDFS files in encryption zones.
- After the job is finished, the RM cancels the Delegation Tokens for the job.
Note: A step not drawn in the above diagram is, RM also tracks each Delegation Token’s expiration time, and renews the Delegation Token when it’s at 90% of the expiration time. Note that Delegation Tokens are tracked in the RM on an individual basis, not Token-Kind basis. This way, tokens with different renewal intervals can all be renewed correctly. The token renewer classes are implemented using Java ServiceLoader, so RM doesn’t have to be aware of the token kinds. For extremely interested readers, the relevant code is in RM’s DelegationTokenRenewer class.
What is this InvalidToken error?
Now we know what Delegation Tokens are and how they are used when running a typical job. But don’t stop here! Let’s look at some typical Delegation Token related error messages from application logs, and figure out what they mean.
Token is expired
Sometimes, the application fails with AuthenticationException, with an InvalidToken exception wrapped inside. The exception message indicates that “token is expired”. Guess why this could happen?
| …. 17/10/22 20:50:12 INFO mapreduce.Job: Job job_1508730362330_0002 failed with state FAILED due to: Application application_1508730362330_0002 failed 2 times due to AM Container for appattempt_1508730362330_0002_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:https://example.cloudera.com:8090/proxy/application_1508730362330_0002/Then, click on links to logs of each attempt. Diagnostics: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730603474, maxDate=1509335403474, sequenceNumber=7, masterKeyId=69) is expired, current time: 2017-10-22 20:50:12,166-0700 expected renewal time: 2017-10-22 20:50:05,518-0700 …. Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730603474, maxDate=1509335403474, sequenceNumber=7, masterKeyId=69) is expired, current time: 2017-10-22 20:50:12,166-0700 expected renewal time: 2017-10-22 20:50:05,518-0700 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) … at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:535) … |
Token can’t be found in cache
Sometimes, the application fails with AuthenticationException, with an InvalidToken exception wrapped inside. The exception message indicates that “token can’t be found in cache”. Guess why this could happen, and what’s the difference with the “token is expired” error?
| …. 17/10/22 20:55:47 INFO mapreduce.Job: Job job_1508730362330_0003 failed with state FAILED due to: Application application_1508730362330_0003 failed 2 times due to AM Container for appattempt_1508730362330_0003_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:https://example.cloudera.com:8090/proxy/application_1508730362330_0003/Then, click on links to logs of each attempt. Diagnostics: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730937041, maxDate=1509335737041, sequenceNumber=8, masterKeyId=73) can’t be found in cache java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730937041, maxDate=1509335737041, sequenceNumber=8, masterKeyId=73) can’t be found in cache at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:294) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:528) at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1448) … at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:784) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:367) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357) … at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=xiao, renewer=yarn, realUser=, issueDate=1508730937041, maxDate=1509335737041, sequenceNumber=8, masterKeyId=73) can’t be found in cache … at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:535) … |
Explanation:
The 2 errors above share the same cause: the token that was used to authenticate has expired, and can no longer be used to authenticate.
The first message was able to tell explicitly that the token has expired, because that token is still stored in the server. Therefore when the server verifies the token, the verification on expiration time fails, throwing the “token is expired” exception.
Now can you guess when the second error could happen? Remember in the ‘Delegation Tokens at the Server-side’ section, we explained the server has a background thread to remove expired tokens. So if the token is being used for authentication after the server’s background thread has removed it, the verification logic on the server cannot find this token. This results in an exception being thrown, saying “token can’t be found”.
The graph below shows the sequence of these events.
Figure 4: Life Cycle of Delegation Token
Note that when a token is explicitly canceled, it will be immediately removed from the store. So for canceled tokens, the error will always be “token can’t be found”.
To further debug these errors, grepping the client logs as well as the server logs for the specific token sequence number (“sequenceNumber=7” or “sequenceNumber=8” in the above examples) is necessary. You should be able to see events related to token creations, renewals (if any), cancelations (if any) in the server logs.
Long-running Applications
At this point, you know all the basics about Hadoop Delegation Tokens. But there is one missing piece beyond it: we know that Delegations Tokens cannot be renewed beyond their max lifetimes, so what happens to applications that do need to run longer than the max lifetime?
The bad news is that Hadoop does not have a unified way for all applications to do this, so there does not exist a magic configuration which any application can turn on to let it ‘just work’. But this is still possible. Good news for Spark-submitted applications is, Spark has implemented these magical parameters. Spark get the Delegation Tokens and uses them for authentication, similar to what we described in the earlier sections. However, Spark does not renew the tokens, and instead just gets a new token when it’s about to expire. This allows the application to run forever. The relevant code is here. Note that this requires giving a Kerberos keytab to your Spark application.
But what if you are implementing a long-running service application, and want the tokens handled explicitly by the application? This would involve two parts: renew the tokens until the max lifetime; handle the token replacement after max lifetime.
Note that this is not a usual practice, and is only recommended if you’re implementing a service.
Implementing a Token Renewer
Let’s first study how token renewal should be done. The best way is to study YARN RM’s DelegationTokenRenewer code.
Some key points from that class are:
- It is a single class to manage all the tokens. Internally it has a thread pool to renew tokens, and another thread to cancel tokens. Renew happens at 90% of the expiration time. Cancel has a small delay (30 seconds) to prevent races.
- Expiration time for each token is managed separately. The token’s expiration time is programmatically retrieved by calling the token’s renew() API. The return value of that API is the expiration time.
dttr.expirationDate =
UserGroupInformation.getLoginUser().doAs(
new PrivilegedExceptionAction<Long>() {
@Override
public Long run() throws Exception {
return dttr.token.renew(dttr.conf);
}
});
This is another reason why YARN RM renews a token immediately after receiving it – to know when it will expire.
3. The max lifetime of each token is retrieved by decoding the token’s identifier, and calling its getMaxDate() API. Other fields in the identifier can also be obtained by calling similar APIs.
if (token.getKind().equals(HDFS_DELEGATION_KIND)) {
try {
AbstractDelegationTokenIdentifier identifier =
(AbstractDelegationTokenIdentifier) token.decodeIdentifier();
maxDate = identifier.getMaxDate();
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
4. Because of 2 and 3, no configuration needs to be read to determine the renew interval and max lifetime. The server’s configuration may change from time to time. The client should not depend on it, and do not have a way to know it.
Handling Tokens after Max Lifetime
The token renewer will only renew the token until its max lifetime. After max lifetime, the job will fail. If your application is long-running, you should consider either utilizing the mechanisms described in YARN documentation about long-lived services, or add logic to your own delegation token renewer class, to retrieve new delegation tokens when the existing ones are about to reach max lifetime.
Other Ways Tokens Are Used
Congratulations! You have now finished reading the concepts and details about Delegation Tokens. There are a few related aspects not covered by this blog post. Below is a brief introduction of them.
Delegation Tokens in other services: Services such as Apache Oozie, Apache Hive, and Apache Hadoop’s YARN RM also utilize Delegation Tokens for authentication.
Block Access Token: HDFS clients access a file by first contacting the NameNode, to get the block locations of a specific file, then access the blocks directly on the DataNode. File permission checking takes place in the NameNode. But for the subsequent data block accesses on DataNodes, authorization checks are required as well. Block Access Tokens are used for this purpose. They are issued by HDFS NameNode to the client, and then passed to DataNode by the client. Block Access Token has a short lifetime (10 hours by default) and can not be renewed. If Block Access Token expires, the client has to request a new one.
Authentication Token: We have covered exclusively Delegation Tokens. Hadoop also has the concept of an Authentication Token, which is designed to be an even cheaper and more scalable authentication mechanism. It acts like a cookie between the server and the client. Authentication Tokens are granted by the server, cannot be renewed or used to impersonate others. And unlike Delegation Tokens, don’t need to be individually stored by the server. You should not need to explicitly code against Authentication Token.
Conclusion
Delegation Tokens play an important role in the Hadoop ecosystem. You should now understand the purpose of Delegation Tokens, how they are used, and why they are used this way. This knowledge is essential when writing and debugging applications.
Appendices
Appendix A. Configurations at the server-side.
Below is the table of configurations related to Delegation Tokens. Please see Delegation Tokens at the Server-side for explanations of these properties.
| Property | Configuration Name in HDFS |
Default Value in HDFS |
Configuration Name in KMS |
Default Value in KMS |
| Renew Interval | dfs.namenode.delegation.token.renew- interval |
86400000 (1 day) |
hadoop.kms.authentication.delegation-token.renew- interval.sec |
86400 (1 day) |
| Max Lifetime | dfs.namenode.delegation.token.max- lifetime |
604800000 (7 days) |
hadoop.kms.authentication.delegation-token.max- lifetime.sec |
604800 (7 days) |
| Interval of background removal of expired Tokens |
Not configurable | 3600000 (1 hour) |
hadoop.kms.authentication.delegation-token.removal- scan-interval.sec |
3600 (1 hour) |
| Master Key Rolling Interval | dfs.namenode.delegation.key.update- interval |
86400000 (1 day) |
hadoop.kms.authentication.delegation-token.update- interval.sec |
86400 (1 day) |
Table 4: Configuration Properties and Default Values for HDFS and KMS
Appendix B. Example Code of Authenticating with Delegation Tokens.
One concept to understand before looking at the code below is the UserGroupInformation (UGI) class. UGI is Hadoop’s public API about coding against authentication. It is used in the code examples below, and also appeared in some of the exception stack traces earlier.
GetFileStatus is used as an example of using UGIs to access HDFS. For details, see FileSystem class javadoc.
UserGroupInformation tokenUGI = UserGroupInformation.createRemoteUser("user_name");
UserGroupInformation kerberosUGI = UserGroupInformation.loginUserFromKeytabAndReturnUGI("principal_in_keytab", "path_to_keytab_file");
Credentials creds = new Credentials();
kerberosUGI.doAs((PrivilegedExceptionAction<Void>) () -> {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.getFileStatus(filePath); // ← kerberosUGI can authenticate via Kerberos
// get delegation tokens, add them to the provided credentials. set renewer to ‘yarn’
Token<?>[] newTokens = fs.addDelegationTokens("yarn", creds);
// Add all the credentials to the UGI object
tokenUGI.addCredentials(creds);
// Alternatively, you can add the tokens to UGI one by one.
// This is functionally the same as the above, but you have the option to log each token acquired.
for (Token<?> token : newTokens) {
tokenUGI.addToken(token);
}
return null;
});
Note that the addDelegationTokens RPC call is invoked with Kerberos authentication. Otherwise, it would result in an IOException thrown, with text saying “Delegation Token can be issued only with kerberos or web authentication”.
Now, we can use the acquired delegation token to access HDFS.
tokenUGI.doAs((PrivilegedExceptionAction<Void>) () -> {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.getFileStatus(filePath); // ← tokenUGI can authenticate via Delegation Token
return null;
});