最近有一個項目,需要將爬蟲獲取的衆多網頁解析出來,並將內容插入hbase,考慮採用mapreduce來實現。由於一個html文件最大不過幾M,所以一個文件將會交給一個map處理,mapreduce中最常見的是按行讀取文本文件,而我們需要的是一次讀取整個文件內容,然後在map方法中用jsoup解析內容。現將實現一次讀取整個文件內容的代碼貼出來。
1、定義一個FileInputFormat類
public class WholeFileInputFormat extends FileInputFormat<Text,Text>{
@Override
public RecordReader<Text, Text> createRecordReader(InputSplit arg0, TaskAttemptContext arg1) throws IOException,
InterruptedException {
// TODO Auto-generated method stub
RecordReader<Text,Text> recordReader = new WholeFileRecordReader();
return recordReader;
}
}
2、自定義RecordReader方法
public class WholeFileRecordReader extends RecordReader<Text,Text>{
private FileSplit fileSplit;
private JobContext jobContext;
private Text currentKey = new Text();
private Text currentValue = new Text();
private boolean finishConverting = false;
@Override
public void close() throws IOException {
// TODO Auto-generated method stub
}
@Override
public Text getCurrentKey() throws IOException, InterruptedException {
// TODO Auto-generated method stub
return currentKey;
}
@Override
public Text getCurrentValue() throws IOException,
InterruptedException {
// TODO Auto-generated method stub
return currentValue;
}
@Override
public float getProgress() throws IOException, InterruptedException {
// TODO Auto-generated method stub
float progress = 0;
if(finishConverting){
progress = 1;
}
return progress;
}
@Override
public void initialize(InputSplit arg0, TaskAttemptContext arg1)
throws IOException, InterruptedException {
this.fileSplit = (FileSplit) arg0;
this.jobContext = arg1;
String filename = fileSplit.getPath().getName();
this.currentKey = new Text(filename);
}
@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
// TODO Auto-generated method stub
if(!finishConverting){
int len = (int)fileSplit.getLength();
// byte[] content = new byte[len];
Path file = fileSplit.getPath();
FileSystem fs = file.getFileSystem(jobContext.getConfiguration());
FSDataInputStream in = fs.open(file);
BufferedReader br = new BufferedReader(new InputStreamReader(in,"gbk"));
// BufferedReader br = new BufferedReader(new InputStreamReader(in,"utf-8"));
String line="";
String total="";
while((line= br.readLine())!= null){
total =total+line+"\n";
}
br.close();
in.close();
fs.close();
currentValue = new Text(total);
finishConverting = true;
return true;
}
return false;
}
}
實現RecordReader接口,最核心的就是處理好迭代多行文本的內容的邏輯,每次迭代通過調用nextKeyValue()方法來判斷是否還有可讀的文本行,直接設置當前的Key和Value,分別在方法getCurrentKey()和getCurrentValue()中返回對應的值。