文章轉自：原創： 任航 若澤大數據
問題 :
HDFS啓動後一直處於安全狀態
解決過程：
1.查看hadoop namenode的啓動日誌
1STARTUP_MSG: Starting NameNode
2STARTUP_MSG: host = hadoop001/192.168.137.141
3STARTUP_MSG: args = []
4STARTUP_MSG: version = 2.6.0-cdh5.7.0
5STARTUP_MSG: classpath = ............
6The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
72019-01-16 21:23:02,807 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
82019-01-16 21:23:02,807 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
92019-01-16 21:23:32,808 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
102019-01-16 21:23:32,808 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
112019-01-16 21:23:48,686 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.name
12node.SafeModeException: Log not rolled. Name node is in safe mode.
13The reported blocks 549 needs additional 4 blocks to reach the threshold 0.9990 of total blocks 553.
14The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
152019-01-16 21:23:48,687 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog from 19
162.168.137.141:53271 Call#84 Retry#0: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled. Name node is in safe mode.
17The reported blocks 549 needs additional 4 blocks to reach the threshold 0.9990 of total blocks 553.
18The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
192019-01-16 21:23:51,355 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.name
20node.SafeModeException: Cannot create directory /tmp/hive/hadoop/b8d6b3ec-3b4b-4e3c-8b9e-6d8d328203aa. Name node is in safe mode.
21The reported blocks 549 needs additional 4 blocks to reach the threshold 0.9990 of total blocks 553.
22The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
232019-01-16 21:23:51,355 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from 192.168.137.141:
2453272 Call#4 Retry#0: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/hadoop/b8d6b3ec-3b4b-4e3c-8b9e-6d8d328203aa. Name node
25is in safe mode.
26The reported blocks 549 needs additional 4 blocks to reach the threshold 0.9990 of total blocks 553.
27The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
282019-01-16 21:24:02,809 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds
292019-01-16 21:24:02,811 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 2 millisecond(s).
302019-01-16 21:24:30,425 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing replication queues
發現是因爲缺少blocks，block數量沒有達到所有塊的0.9990的閾值（needs additional 4 blocks to reach the threshold 0.9990 of total blocks 553）
爲什麼會丟塊呢？
2.查看hadoop namenode的啓動日誌
1/************************************************************
2STARTUP_MSG: Starting DataNode
3STARTUP_MSG: host = hadoop001/192.168.137.141
4STARTUP_MSG: args = []
5STARTUP_MSG: version = 2.6.0-cdh5.7.0
6STARTUP_MSG: classpath = ......
72019-01-16 20:14:33,776 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled for /home/hadoop/app/tmp/dfs/data/current/BP-848574762-192.168.137.141-152
88737264517
92019-01-16 20:14:33,780 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=1759064178;bpid=BP-848574762-192.168.137.141-1528737264517;lv=-56;
10nsInfo=lv=-60;cid=CID-2ccbdca6-0244-4b19-8107-f20a1a74cfe0;nsid=1759064178;c=0;bpid=BP-848574762-192.168.137.141-1528737264517;dnuuid=3d4c88f5-6ffd-4b12-a90f-bba9b8e65b7
114
122019-01-16 20:14:34,167 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-23637e15-c56f-4cc7-aecf-f1a2288cb71e
132019-01-16 20:14:34,167 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /home/hadoop/app/tmp/dfs/data/current, StorageType: DISK
142019-01-16 20:14:34,251 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean
152019-01-16 20:14:34,262 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Volume reference is released.
162019-01-16 20:14:34,263 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-848574762-192.168.137.141-1528737264517
172019-01-16 20:14:34,277 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-848574762-192.168.137.141-1528737264517 on volume /home/hadoop/app/tmp/dfs/data/current...
182019-01-16 20:14:34,392 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Cached dfsUsed found for /home/hadoop/app/tmp/dfs/data/current/BP-848574762-192.168.137.141-1528737264517/current: 739151872
192019-01-16 20:14:34,408 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-848574762-192.168.137.141-1528737264517 on /home/hadoop/app/tmp/dfs/data/current: 128ms
202019-01-16 20:14:34,409 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-848574762-192.168.137.141-1528737264517: 146ms
212019-01-16 20:14:34,416 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-848574762-192.168.137.141-1528737264517 on volume /home/hadoop/app/tmp/dfs/data/current...
222019-01-16 20:14:34,980 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-848574762-192.168.137.141-1528737264517 on volume /home/hadoop/app/tmp/dfs/data/current: 564ms
232019-01-16 20:14:34,980 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map: 568ms
242019-01-16 20:14:36,271 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/home/hadoop/app/tmp/dfs/data, DS-23637e15-c56f-4cc7-aecf-f1a2288cb71e): no suitable block pools found to scan. Waiting 846698096 ms.
252019-01-16 20:14:36,280 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1547658816280ms with interval of 21600000ms
262019-01-16 20:14:36,295 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-848574762-192.168.137.141-1528737264517 (Datanode Uuid null) service to hadoop001/192.168.137.141:9000 beginning handshake with NN
272019-01-16 20:14:36,491 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-848574762-192.168.137.141-1528737264517 (Datanode Uuid null) service to hadoop001/192.168.137.141:9000 successfully registered with NN
282019-01-16 20:14:36,491 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode hadoop001/192.168.137.141:9000 using BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 0msec; heartBeatInterval=3000
datanode的啓動日誌沒有發現問題
下圖爲與datanode檢查相關的參數設置
查看官網dfs.namenode.safemode.threshold-pct參數的默認值爲0.999f
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
3.使用hdfs fsck /命令查看當前的hdfs的塊的狀態
1[hadoop@hadoop001 hadoop]$ hdfs fsck /
2Connecting to namenode via http://hadoop001:50070
3FSCK started by hadoop (auth:SIMPLE) from /192.168.137.141 for path / at Wed Jan 16 22:39:38 CST 2019
4...........................
5/out/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744061_3237. Target Replicas is 3 but found 1 replica(s).
6.
7/out/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744060_3236. Target Replicas is 3 but found 1 replica(s).
8......
9/spark/checkpointdata/checkpoint-1543419775000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745968_5150. Target Replicas is 3 but found 1 replica(s).
10.
11/spark/checkpointdata/checkpoint-1543419775000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745967_5149. Target Replicas is 3 but found 1 replica(s).
12.
13/spark/checkpointdata/checkpoint-1543419780000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745970_5152. Target Replicas is 3 but found 1 replica(s).
14.
15/spark/checkpointdata/checkpoint-1543419780000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745969_5151. Target Replicas is 3 but found 1 replica(s).
16.
17/spark/checkpointdata/checkpoint-1543419785000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745972_5154. Target Replicas is 3 but found 1 replica(s).
18.
19/spark/checkpointdata/checkpoint-1543419785000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745971_5153. Target Replicas is 3 but found 1 replica(s).
20.
21/spark/checkpointdata/checkpoint-1543419790000: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745974
22/spark/checkpointdata/checkpoint-1543419790000: MISSING 1 blocks of total size 4924 B..
23/spark/checkpointdata/checkpoint-1543419790000.bk: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745973
24/spark/checkpointdata/checkpoint-1543419790000.bk: MISSING 1 blocks of total size 4943 B..
25/spark/checkpointdata/checkpoint-1543419795000: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745976
26/spark/checkpointdata/checkpoint-1543419795000: MISSING 1 blocks of total size 4924 B..
27/spark/checkpointdata/checkpoint-1543419795000.bk: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745975
28/spark/checkpointdata/checkpoint-1543419795000.bk: MISSING 1 blocks of total size 4946 B..
29/spark/checkpointdata/receivedBlockMetadata/log-1543419757424-1543419817424: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745960_5159. Target Replicas is 3 but found 1 replica(s).
30.
31/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/_partitioner: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744412_3588. Target Replicas is 3 but found 1 replica(s).
32.
33/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744410_3586. Target Replicas is 3 but found 1 replica(s).
34.
35/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744411_3587. Target Replicas is 3 but found 1 replica(s).
36.
37/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/_partitioner: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744419_3595. Target Replicas is 3 but found 1 replica(s).
38.
39/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744417_3593. Target Replicas is 3 but found 1 replica(s).
40.
41/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744418_3594. Target Replicas is 3 but found 1 replica(s).
42.
43/streaming/checkpoint/checkpoint-1538092435000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744405_3581. Target Replicas is 3 but found 1 replica(s).
44.
45/streaming/checkpoint/checkpoint-1538092435000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744401_3577. Target Replicas is 3 but found 1 replica(s).
46.
47/streaming/checkpoint/checkpoint-1538092440000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744407_3583. Target Replicas is 3 but found 1 replica(s).
48.
49/streaming/checkpoint/checkpoint-1538092440000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744406_3582. Target Replicas is 3 but found 1 replica(s).
50.
51/streaming/checkpoint/checkpoint-1538092445000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744413_3589. Target Replicas is 3 but found 1 replica(s).
52.
53/streaming/checkpoint/checkpoint-1538092445000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744409_3585. Target Replicas is 3 but found 1 replica(s).
54.
55/streaming/checkpoint/checkpoint-1538092450000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744415_3591. Target Replicas is 3 but found 1 replica(s).
56.
57/streaming/checkpoint/checkpoint-1538092450000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744414_3590. Target Replicas is 3 but found 1 replica(s).
58.
59/streaming/checkpoint/checkpoint-1538092455000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744420_3596. Target Replicas is 3 but found 1 replica(s).
60.
61/streaming/checkpoint/checkpoint-1538092455000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744416_3592. Target Replicas is 3 but found 1 replica(s).
62.
63/streaming/checkpoint/receivedBlockMetadata/log-1538092380422-1538092440422: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744365_3541. Target Replicas is 3 but found 1 replica(s).
64.
65/streaming/checkpoint/receivedBlockMetadata/log-1538092445001-1538092505001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744408_3597. Target Replicas is 3 but found 1 replica(s).
66......................................
67....................................................................................................
68....................................................................................................
69....................................................................................................
70....................................................................................................
71.......................................................................Status: CORRUPT
72 Total size: 730163442 B
73 Total dirs: 113
74 Total files: 571
75 Total symlinks: 0
76 Total blocks (validated): 553 (avg. block size 1320367 B)
77 ********************************
78 CORRUPT FILES: 4
79 MISSING BLOCKS: 4
80 MISSING SIZE: 19737 B
81 CORRUPT BLOCKS: 4
82 ********************************
83 Minimally replicated blocks: 549 (99.27667 %)
84 Over-replicated blocks: 0 (0.0 %)
85 Under-replicated blocks: 27 (4.882459 %)
86 Mis-replicated blocks: 0 (0.0 %)
87 Default replication factor: 1
88 Average block replication: 0.99276674
89 Corrupt blocks: 4
90 Missing replicas: 54 (8.780488 %)
91 Number of data-nodes: 1
92 Number of racks: 1
93FSCK ended at Wed Jan 16 22:39:39 CST 2019 in 626 milliseconds
94The filesystem under path '/' is CORRUPT
發現其中有4個blocks損壞
4.執行命令hdfs fsck / -delete，即刪除破損的文件（注意: 這種方式會出現數據丟失，損壞的文件會被刪掉）
1[hadoop@hadoop001 hadoop]$ hdfs fsck / -delete
2Connecting to namenode via http://hadoop001:50070
3FSCK started by hadoop (auth:SIMPLE) from /192.168.137.141 for path / at Wed Jan 16 22:44:36 CST 2019
4...........................
5/out/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744061_3237. Target Replicas is 3 but found 1 replica(s).
6.
7/out/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744060_3236. Target Replicas is 3 but found 1 replica(s).
8......
9/spark/checkpointdata/checkpoint-1543419775000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745968_5150. Target Replicas is 3 but found 1 replica(s).
10.
11/spark/checkpointdata/checkpoint-1543419775000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745967_5149. Target Replicas is 3 but found 1 replica(s).
12.
13/spark/checkpointdata/checkpoint-1543419780000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745970_5152. Target Replicas is 3 but found 1 replica(s).
14.
15/spark/checkpointdata/checkpoint-1543419780000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745969_5151. Target Replicas is 3 but found 1 replica(s).
16.
17/spark/checkpointdata/checkpoint-1543419785000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745972_5154. Target Replicas is 3 but found 1 replica(s).
18.
19/spark/checkpointdata/checkpoint-1543419785000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745971_5153. Target Replicas is 3 but found 1 replica(s).
20.
21/spark/checkpointdata/checkpoint-1543419790000: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745974
22/spark/checkpointdata/checkpoint-1543419790000: MISSING 1 blocks of total size 4924 B..
23/spark/checkpointdata/checkpoint-1543419790000.bk: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745973
24/spark/checkpointdata/checkpoint-1543419790000.bk: MISSING 1 blocks of total size 4943 B..
25/spark/checkpointdata/checkpoint-1543419795000: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745976
26/spark/checkpointdata/checkpoint-1543419795000: MISSING 1 blocks of total size 4924 B..
27/spark/checkpointdata/checkpoint-1543419795000.bk: CORRUPT blockpool BP-848574762-192.168.137.141-1528737264517 block blk_1073745975
28/spark/checkpointdata/checkpoint-1543419795000.bk: MISSING 1 blocks of total size 4946 B..
29/spark/checkpointdata/receivedBlockMetadata/log-1543419757424-1543419817424: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745960_5159. Target Replicas is 3 but found 1 replica(s).
30.
31/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/_partitioner: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744412_3588. Target Replicas is 3 but found 1 replica(s).
32.
33/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744410_3586. Target Replicas is 3 but found 1 replica(s).
34.
35/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744411_3587. Target Replicas is 3 but found 1 replica(s).
36.
37/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/_partitioner: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744419_3595. Target Replicas is 3 but found 1 replica(s).
38.
39/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744417_3593. Target Replicas is 3 but found 1 replica(s).
40.
41/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744418_3594. Target Replicas is 3 but found 1 replica(s).
42.
43/streaming/checkpoint/checkpoint-1538092435000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744405_3581. Target Replicas is 3 but found 1 replica(s).
44.
45/streaming/checkpoint/checkpoint-1538092435000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744401_3577. Target Replicas is 3 but found 1 replica(s).
46.
47/streaming/checkpoint/checkpoint-1538092440000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744407_3583. Target Replicas is 3 but found 1 replica(s).
48.
49/streaming/checkpoint/checkpoint-1538092440000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744406_3582. Target Replicas is 3 but found 1 replica(s).
50.
51/streaming/checkpoint/checkpoint-1538092445000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744413_3589. Target Replicas is 3 but found 1 replica(s).
52.
53/streaming/checkpoint/checkpoint-1538092445000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744409_3585. Target Replicas is 3 but found 1 replica(s).
54.
55/streaming/checkpoint/checkpoint-1538092450000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744415_3591. Target Replicas is 3 but found 1 replica(s).
56.
57/streaming/checkpoint/checkpoint-1538092450000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744414_3590. Target Replicas is 3 but found 1 replica(s).
58.
59/streaming/checkpoint/checkpoint-1538092455000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744420_3596. Target Replicas is 3 but found 1 replica(s).
60.
61/streaming/checkpoint/checkpoint-1538092455000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744416_3592. Target Replicas is 3 but found 1 replica(s).
62.
63/streaming/checkpoint/receivedBlockMetadata/log-1538092380422-1538092440422: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744365_3541. Target Replicas is 3 but found 1 replica(s).
64.
65/streaming/checkpoint/receivedBlockMetadata/log-1538092445001-1538092505001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744408_3597. Target Replicas is 3 but found 1 replica(s).
66......................................
67....................................................................................................
68....................................................................................................
69....................................................................................................
70....................................................................................................
71.......................................................................Status: CORRUPT
72 Total size: 730163442 B
73 Total dirs: 113
74 Total files: 571
75 Total symlinks: 0
76 Total blocks (validated): 553 (avg. block size 1320367 B)
77 ********************************
78 CORRUPT FILES: 4
79 MISSING BLOCKS: 4
80 MISSING SIZE: 19737 B
81 CORRUPT BLOCKS: 4
82 ********************************
83 Minimally replicated blocks: 549 (99.27667 %)
84 Over-replicated blocks: 0 (0.0 %)
85 Under-replicated blocks: 27 (4.882459 %)
86 Mis-replicated blocks: 0 (0.0 %)
87 Default replication factor: 1
88 Average block replication: 0.99276674
89 Corrupt blocks: 4
90 Missing replicas: 54 (8.780488 %)
91 Number of data-nodes: 1
92 Number of racks: 1
93FSCK ended at Wed Jan 16 22:44:36 CST 2019 in 308 milliseconds
94The filesystem under path '/' is CORRUPT
5.再次執行命令hdfs fsck /
1[hadoop@hadoop001 hadoop]$ hdfs fsck /
2Connecting to namenode via http://hadoop001:50070
3FSCK started by hadoop (auth:SIMPLE) from /192.168.137.141 for path / at Wed Jan 16 22:45:04 CST 2019
4...........................
5/out/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744061_3237. Target Replicas is 3 but found 1 replica(s).
6.
7/out/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744060_3236. Target Replicas is 3 but found 1 replica(s).
8......
9/spark/checkpointdata/checkpoint-1543419775000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745968_5150. Target Replicas is 3 but found 1 replica(s).
10.
11/spark/checkpointdata/checkpoint-1543419775000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745967_5149. Target Replicas is 3 but found 1 replica(s).
12.
13/spark/checkpointdata/checkpoint-1543419780000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745970_5152. Target Replicas is 3 but found 1 replica(s).
14.
15/spark/checkpointdata/checkpoint-1543419780000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745969_5151. Target Replicas is 3 but found 1 replica(s).
16.
17/spark/checkpointdata/checkpoint-1543419785000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745972_5154. Target Replicas is 3 but found 1 replica(s).
18.
19/spark/checkpointdata/checkpoint-1543419785000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745971_5153. Target Replicas is 3 but found 1 replica(s).
20.
21/spark/checkpointdata/receivedBlockMetadata/log-1543419757424-1543419817424: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073745960_5159. Target Replicas is 3 but found 1 replica(s).
22.
23/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/_partitioner: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744412_3588. Target Replicas is 3 but found 1 replica(s).
24.
25/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744410_3586. Target Replicas is 3 but found 1 replica(s).
26.
27/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-167/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744411_3587. Target Replicas is 3 but found 1 replica(s).
28.
29/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/_partitioner: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744419_3595. Target Replicas is 3 but found 1 replica(s).
30.
31/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/part-00000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744417_3593. Target Replicas is 3 but found 1 replica(s).
32.
33/streaming/checkpoint/89d2e5e2-ad1a-497b-ba5a-823ab475cf16/rdd-180/part-00001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744418_3594. Target Replicas is 3 but found 1 replica(s).
34.
35/streaming/checkpoint/checkpoint-1538092435000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744405_3581. Target Replicas is 3 but found 1 replica(s).
36.
37/streaming/checkpoint/checkpoint-1538092435000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744401_3577. Target Replicas is 3 but found 1 replica(s).
38.
39/streaming/checkpoint/checkpoint-1538092440000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744407_3583. Target Replicas is 3 but found 1 replica(s).
40.
41/streaming/checkpoint/checkpoint-1538092440000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744406_3582. Target Replicas is 3 but found 1 replica(s).
42.
43/streaming/checkpoint/checkpoint-1538092445000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744413_3589. Target Replicas is 3 but found 1 replica(s).
44.
45/streaming/checkpoint/checkpoint-1538092445000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744409_3585. Target Replicas is 3 but found 1 replica(s).
46.
47/streaming/checkpoint/checkpoint-1538092450000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744415_3591. Target Replicas is 3 but found 1 replica(s).
48.
49/streaming/checkpoint/checkpoint-1538092450000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744414_3590. Target Replicas is 3 but found 1 replica(s).
50.
51/streaming/checkpoint/checkpoint-1538092455000: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744420_3596. Target Replicas is 3 but found 1 replica(s).
52.
53/streaming/checkpoint/checkpoint-1538092455000.bk: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744416_3592. Target Replicas is 3 but found 1 replica(s).
54.
55/streaming/checkpoint/receivedBlockMetadata/log-1538092380422-1538092440422: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744365_3541. Target Replicas is 3 but found 1 replica(s).
56.
57/streaming/checkpoint/receivedBlockMetadata/log-1538092445001-1538092505001: Under replicated BP-848574762-192.168.137.141-1528737264517:blk_1073744408_3597. Target Replicas is 3 but found 1 replica(s).
58..........................................
59....................................................................................................
60....................................................................................................
61....................................................................................................
62....................................................................................................
63...................................................................Status: HEALTHY
64 Total size: 730143705 B
65 Total dirs: 113
66 Total files: 567
67 Total symlinks: 0
68 Total blocks (validated): 549 (avg. block size 1329952 B)
69 Minimally replicated blocks: 549 (100.0 %)
70 Over-replicated blocks: 0 (0.0 %)
71 Under-replicated blocks: 27 (4.9180326 %)
72 Mis-replicated blocks: 0 (0.0 %)
73 Default replication factor: 1
74 Average block replication: 1.0
75 Corrupt blocks: 0
76 Missing replicas: 54 (8.955224 %)
77 Number of data-nodes: 1
78 Number of racks: 1
79FSCK ended at Wed Jan 16 22:45:04 CST 2019 in 303 milliseconds
80The filesystem under path '/' is HEALTHY
發現損壞的4個blocks已經被刪除，狀態檢查爲HEALTHY