場景:
項目組6個節點中,某一節點硬盤壽命到了,由於當時硬盤做的raid0,沒有備份,更換新硬盤後,啓動datanode遇到如下問題:
1. 更換硬盤前,停止datanode,停止在此節點上應用;
2. 停止機器,更換硬盤
3. 重啓機器,重啓datanode,遇到問題
3.1
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid dfs.datanode.data.dir /data :
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid dfs.datanode.data.dir /data :
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:729)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:505)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:486)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:502)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:140)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:156)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2345)
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2387)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2369)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2261)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2308)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2485)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2509)
2019-03-05 21:01:10,707 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/data/"
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2396)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2369)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2261)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2308)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2485)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2509)
2019-03-05 21:01:10,708 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
原因:
由於更換硬盤,針對/data目錄的權限變成了root,需要對當前用戶賦予權限
解決辦法:
chown -R hadoop:hadoop /data
3.2
2019-03-05 21:25:47,891 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: failed to stat a path component: '/var/run/hdfs-sockets'
2019-03-05 21:25:47,891 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: failed to stat a path component: '/var/run/hdfs-sockets'. error code 2 (No such file or directory)
at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)
at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:189)
at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)
at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:946)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:912)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1111)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:429)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2374)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2261)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2308)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2485)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2509)
2019-03-05 21:25:47,893 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2019-03-05 21:25:47,894 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
原因:
沒有/var/run/hdfs-sockets 這個路徑
解決辦法:
創建這個目錄即可