因爲項目需要，做了一個網絡爬蟲的小DEMO。
爲實現高性能的網絡爬蟲，首先考慮採用APACE的HttpClient進行頁面的採集和解析，HttpClient可以很方便的通過URL獲得遠程內容，例如一個小程序：
CloseableHttpClienthttp client = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://localhost/");
CloseableHttpResponse response = httpclient.execute(httpget);
try
{
HttpEntity entity =response.getEntity();
if (entity != null) {
long len =entity.getContentLength();
if (len != -1 && len <2048) {
System.out.println(EntityUtils.toString(entity));
} else {
// Stream contentout
}
}
}
finally {
response.close();
}還可以做頁面解析和模擬登陸等，功能相當強大。
其次，如果是網絡爬蟲或者網絡採集，可能需要做大量的URL地址收集和分析，所以需要通過NoSQL數據庫來提高執行的效率，Redis、Memcache、BerkeleyDB都是不錯的選擇。這裏選擇了BerkeleyDB數據庫。雖然採用傳統隊列或其他形式可能性能會更高，但會帶來大量的內存消耗，並不一定能找到符合條件的大內存服務器。
然後，對URL地址需要進行過濾，判斷是否是已讀的URL地址，如果已讀就存入已讀數據庫，如果未讀則放入未讀數據庫，有點類似隊列的形式，以此避免重複讀取URL地址。當然更進一步的需要判斷頁面內容是否重複，降低讀取重複頁面的概率。
再然後，對頁面進行解析，提取關鍵內容和URL地址。
最後，爲了保證性能，採用多線程的實現方式，在多服務器的模式下還可以採用分佈式算法來實現更高的性能。
按照上面的思路，寫了一個小程序：
1、部分配置信息，以CrawlConfig來做配置，也可以把這些存儲爲xml文件，這裏採集的是163網站
（1）CrawlConfig.java
……
public class CrawlConfig {
public static final String CRAWL_PATH = "http://www.163.com";
public static final String CRAWL_LIMIT_PATH = "http://www.163.com";
public static final String CRAWL_VISITED_FRONTIER = "d:\\cache\\hevisited";
public static final String CRAWL_UNVISITED_FRONTIER = "d:\\cache\\heunvisited";
public static final String CRAWL_DOWNLOAD_PATH = "d:\\download\\163\\";
public static final int CRAWL_THREAD_NUM = 6;
}（2） CrawlUrl.java作爲URL地址的對象，當然除了URL屬性外還可以存儲其他信息
……
public class CrawlUrl implements Serializable{
private static final long serialVersionUID = 79332323432323L;
public CrawlUrl() {
}
private String oriUrl; //原始url
private String url; //url地址
public String getOriUrl() {
return oriUrl;
}
public void setOriUrl(String oriUrl) {
this.oriUrl = oriUrl;
}
public String getUrl() {
return url;
}
public void setUrl(String url) {
this.url = url;
}
}（3）LinkFilter.java ，作爲URL地址的過濾器
public interface LinkFilter {
public boolean accept(String url);
}2、編寫訪問BerkelyDB的代碼（請先安裝BerkeleyDB，並引入BerkeleyDB的Je包
（1）AbstractFrontier.java
……
public abstract class AbstractFrontier {
private Environment env;
private static String CLASS_CATALOG = "java_class_catalog";
protected StoredClassCatalog javaCatalog;
protected Database catalogdatabase;
protected static Database database = null ;
protected String homeDirectory = null;
public AbstractFrontier(String homeDirectory) throws DatabaseException,
FileNotFoundException {
this.homeDirectory = homeDirectory;
System.out.println("open environment: " + homeDirectory);
//設置環境參數，打開env
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setTransactional(true);
envConfig.setAllowCreate(true);
env = new Environment(new File(homeDirectory), envConfig);
//設置數據庫參數
DatabaseConfig dbConfig = new DatabaseConfig();
dbConfig.setTransactional(true);
dbConfig.setAllowCreate(true);
//打開數據庫
catalogdatabase = env.openDatabase(null, CLASS_CATALOG, dbConfig);
javaCatalog = new StoredClassCatalog(catalogdatabase);
//設置參數
DatabaseConfig dbConfigTe = new DatabaseConfig();
dbConfigTe.setTransactional(true);
dbConfigTe.setAllowCreate(true);
//打開數據庫
database = env.openDatabase(null, "URL", dbConfig);
}
public void close() throws DatabaseException {
database.close();
javaCatalog.close();
env.close();
}
protected abstract void put(Object key, Object value);
protected abstract Object get(Object key);
protected abstract Object delete(Object key);
}（2）Frontier.java
……
public interface Frontier {
public CrawlUrl getNext() throws Exception;
public boolean putUrl(CrawlUrl url) throws Exception;
}（3）考慮到併發的BDBFrontier.java
……
public class BDBFrontier extends AbstractFrontier implements Frontier{
private StoredMap pendingUrisDB = null;
public static int threads = CrawlConfig.CRAWL_THREAD_NUM;
/**
* Creates a new instance of BDBFrontier.
*
* @param homeDirectory
* @throws DatabaseException
* @throws FileNotFoundException
*/
public BDBFrontier(String homeDirectory) throws DatabaseException,
FileNotFoundException {
super(homeDirectory);
EntryBinding keyBinding = new SerialBinding(javaCatalog, String.class);
EntryBinding valueBinding = new SerialBinding(javaCatalog, CrawlUrl.class);
pendingUrisDB = new StoredMap(database, keyBinding, valueBinding, true);
}
/**
*
* clearAll:
* 清除數據庫
*
* @param 參數
* @return void 返回值
* @throws
*
*/
public void clearAll() {
if(!pendingUrisDB.isEmpty())
pendingUrisDB.clear();
}
/**
* 獲得下一條記錄
* @see com.fc.frontier.Frontier#getNext()
*/
@Override
public synchronized CrawlUrl getNext() throws Exception {
CrawlUrl result = null;
while(true) {
if(!pendingUrisDB.isEmpty()) {
Set entrys = pendingUrisDB.entrySet();
Entry<String, CrawlUrl> entry = (Entry<String,
CrawlUrl>) pendingUrisDB.entrySet().iterator().next();
result = entry.getValue(); //下一條記錄
delete(entry.getKey()); //刪除當前記錄
System.out.println("get:" + homeDirectory + entrys);
return result;
}
else {
threads --;
if(threads > 0) {
wait();
threads ++;
}
else {
notifyAll();
return null;
}
}
}
}
/**
* 存入url
* @see com.fc.frontier.Frontier#putUrl(com.fc.CrawlUrl)
*/
@Override
public synchronized boolean putUrl(CrawlUrl url) throws Exception {
if(url.getOriUrl() != null && !url.getOriUrl().equals("")
&& !pendingUrisDB.containsKey(url.getOriUrl()))
{
Set entrys = pendingUrisDB.entrySet();
put(url.getOriUrl(), url);
notifyAll();
System.out.println("put:" + homeDirectory + entrys);
return true;
}
return false;
}
public boolean contains(Object key) {
if(pendingUrisDB.containsKey(key))
return true;
return false;
}
/**
* 存入數據庫
* @see com.fc.frontier.AbstractFrontier#put(java.lang.Object, java.lang.Object)
*/
@Override
protected synchronized void put(Object key, Object value) {
pendingUrisDB.put(key, value);
}
/**
* 從數據庫取出
* @see com.fc.frontier.AbstractFrontier#get(java.lang.Object)
*/
@Override
protected synchronized Object get(Object key) {
return pendingUrisDB.get(key);
}
/**
* 刪除
* @see com.fc.frontier.AbstractFrontier#delete(java.lang.Object)
*/
@Override
protected synchronized Object delete(Object key) {
return pendingUrisDB.remove(key);
}
/**
*
* calculateUrl:
* 對Url進行計算，可以用壓縮算法
*
* @param 參數
* @return String 返回值
* @throws
*
*/
private String calculateUrl(String url) {
return url;
}
public static void main(String[] strs) {
try {
BDBFrontier bdbFrontier = new BDBFrontier("d:\\cache");
CrawlUrl url = new CrawlUrl();
url.setOriUrl("http://www.163.com");
bdbFrontier.putUrl(url);
System.out.println(((CrawlUrl)bdbFrontier.getNext()).getOriUrl());
bdbFrontier.close();
}catch(Exception e) {
e.printStackTrace();
}
}
}3、核心部分：包括了URL獲取、頁面解析、頁面下載，頁面的解析和下載會比較消耗時間。
（1）RetievePage.java，實現了URL訪問和頁面的下載
……
public class RetrievePage {
private static String USER_AGENT = "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1; QQDownload 1.7; .NET CLR 1.1.4322; CIBA; .NET CLR
2.0.50727";
private static String DEFAULT_CHARSET = "GB2312,utf-8;q=0.7,*;q=0.7";
private static String ACCEPT = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
/**
* 下載文件
* @param path
* @return
* @throws Exception
* @throws IOException
*/
public static boolean downloadPage(String path) throws Exception,IOException
{
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(path);
httpget.addHeader("Accept-Charset", DEFAULT_CHARSET);
// httpget.addHeader("Host", host);
httpget.addHeader("Accept", ACCEPT);
httpget.addHeader("User-Agent", USER_AGENT);
RequestConfig requestConfig = RequestConfig.custom() //設置超時
.setSocketTimeout(1000)
.setConnectTimeout(1000)
.build();
httpget.setConfig(requestConfig);
CloseableHttpResponse response = httpclient.execute(httpget);
try {
HttpEntity entity = response.getEntity();
StatusLine statusLine = response.getStatusLine();
if(statusLine.getStatusCode() == HttpStatus.SC_MOVED_PERMANENTLY || //如果是轉移
statusLine.getStatusCode() == HttpStatus.SC_MOVED_TEMPORARILY ||
statusLine.getStatusCode() == HttpStatus.SC_SEE_OTHER ||
statusLine.getStatusCode() == HttpStatus.SC_TEMPORARY_REDIRECT)
{
Header header = httpget.getFirstHeader("location");
if(header != null){
String newUrl = header.getValue();
if(newUrl == null || newUrl.equals(""))
{
newUrl = "/";
HttpGet redirect = new HttpGet(newUrl);
}
}
}
if(statusLine.getStatusCode() == HttpStatus.SC_OK) { //成功訪問
if (entity == null) {
throw new ClientProtocolException("Response contains no content");
}
else {
InputStream instream = entity.getContent();
String filename = getFilenameByUrl(path,entity.getContentType().getValue());
OutputStream outstream = new
FileOutputStream(CrawlConfig.CRAWL_DOWNLOAD_PATH +
filename); //存儲到磁盤
try {
//System.out.println(convertStreamToString(instream));
int tempByte = -1;
while((tempByte = instream.read())>0)
{
outstream.write(tempByte);
}
return true;
}
catch(Exception e){
e.printStackTrace();
return false;
} finally {
if(instream != null)
{
instream.close();
}
if(outstream != null)
{
outstream.close();
}
}
}
}
return false;
}finally {
response.close();
}
}
public static String getFilenameByUrl(String url, String contentType) {
url = url.substring(7);
if(contentType.indexOf("html") != -1) {
url = url.replaceAll("[\\?/:*|<>\"]","_") + ".html";
return url;
}
else {
url = url.replaceAll("[\\?/:*|<>\"]","_") + contentType.substring(contentType.lastIndexOf('/') + 1);
return url;
}
}
/**
* 轉換流數據爲字符串
* @param is
* @return
*/
public static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "/n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
}
return sb.toString();
}
public static void main(String[] args)
{
try{
System.out.println("下載開始");
RetrievePage.downloadPage("http://www.baidu.com");
System.out.println("下載結束");
}
catch(HttpException e){
e.printStackTrace();
}
catch(IOException e)
{
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
}（2）HtmlParserTool.java，實現頁面的解析，提取URL地址
……
public class HtmlParserTool {
public static Set<String> extractLinks(String url, LinkFilter filter){
Set<String> links = new HashSet<String>();
try {
Parser parser = new Parser(url);
parser.setEncoding("gb2312");
NodeFilter frameFilter = new NodeFilter() { //過濾節點
public boolean accept(Node node) {
if(node.getText().startsWith("frame src=")) {
return true;
}
else {
return false;
}
}
};
OrFilter linkFilter = new OrFilter(new NodeClassFilter(LinkTag.class), frameFilter);
NodeList list = parser.extractAllNodesThatMatch(linkFilter); //獲取所有合適的節點
for(int i = 0; i <list.size();i++)
{
Node tag = list.elementAt(i);
if(tag instanceof LinkTag) { //鏈接文字
LinkTag linkTag = (LinkTag) tag;
String linkUrl = linkTag.getLink();//url
String text = linkTag.getLinkText();//鏈接文字
System.out.println(linkUrl + "**********" + text);
if(filter.accept(linkUrl))
links.add(linkUrl);
}
else if (tag instanceof ImageTag) //<img> 標籤 //鏈接圖片
{
ImageTag image = (ImageTag) list.elementAt(i);
System.out.print(image.getImageURL() + "********");//圖片地址
System.out.println(image.getText());//圖片文字
if(filter.accept(image.getImageURL()))
links.add(image.getImageURL());
}
else//<frame> 標籤
{
//提取 frame 裏 src 屬性的鏈接如 <frame src="test.html"/>
String frame = tag.getText();
int start = frame.indexOf("src=");
frame = frame.substring(start);
int end = frame.indexOf(" ");
if (end == -1)
end = frame.indexOf(">");
frame = frame.substring(5, end - 1);
System.out.println(frame);
if(filter.accept(frame))
links.add(frame);
}
}
return links;
} catch (ParserException e) {
e.printStackTrace();
return null;
}
}
}（3）MyCrawler.java實現頁面的採集，這裏採用了寬度優先的採集規則，當然更復雜的考慮這裏還要設置深度，這裏主要採用域名前綴作爲過濾條件。另外多線程環境下，需要考慮數據的同步問題。
……
public class MyCrawler {
public static BDBFrontier visitedFrontier;
public static BDBFrontier unvisitedFrontier;
private static int num = 0;
public MyCrawler() {
try{
if(visitedFrontier == null){
visitedFrontier = new BDBFrontier(CrawlConfig.CRAWL_VISITED_FRONTIER); //採用Nosql數據庫存儲訪問地址方式
visitedFrontier.clearAll();
}
if(unvisitedFrontier == null) {
unvisitedFrontier = new BDBFrontier(CrawlConfig.CRAWL_UNVISITED_FRONTIER);
unvisitedFrontier.clearAll();
}
}catch(Exception e) {
e.printStackTrace();
}
}
private void initCrawlerWithSeeds(String[] seeds) {
synchronized (this) {
try {
for(int i = 0;i<seeds.length;i++){
CrawlUrl url = new CrawlUrl(); //採用berkeleyDB形式
url.setOriUrl(seeds[i]);
unvisitedFrontier.putUrl(url);
}
} catch(Exception e) {
e.printStackTrace();
}
}
}
public void crawling(String[] seeds, int threadId) {
try {
LinkFilter filter = new LinkFilter() {
@Override
public boolean accept(String url) {
Pattern pattern = Pattern.compile("^((https|http|ftp|rtsp|mms)?://)"
+ "+(([0-9a-z_!~*'().&=+$%-]+: )?[0-9a-z_!~*'().&=+$%-]+@)?"
+ "(([0-9]{1,3}\\.){3}[0-9]{1,3}"
+ "|"
+ "([0-9a-z_!~*'()-]+\\.)*"
+ "([0-9a-z][0-9a-z-]{0,61})?[0-9a-z]\\."
+ "[a-z]{2,6})"
+ "(:[0-9]{1,4})?"
+ "((/?)|"
+ "(/[0-9a-z_!~*'().;?:@&=+$,%#-]+)+/?)$");
Matcher matcher = pattern.matcher(url);
boolean isMatch= matcher.matches();
if(isMatch && url.startsWith(CrawlConfig.CRAWL_LIMIT_PATH)) {
return true;
}
else {
return false;
}
}
};
initCrawlerWithSeeds(seeds);
//採用berkeleyDB方式存儲
CrawlUrl visitedCrawlUrl = (CrawlUrl)unvisitedFrontier.getNext();
//visitedFrontier.putUrl(visitedCrawlUrl);
do{
System.out.println("線程：" + threadId);
if(visitedCrawlUrl == null) {
continue;
}
String visitedUrl = visitedCrawlUrl.getOriUrl();
if(visitedFrontier.contains(visitedUrl)) { //同步數據
visitedCrawlUrl = (CrawlUrl)unvisitedFrontier.getNext();
continue;
}
visitedFrontier.putUrl(visitedCrawlUrl);
if(null == visitedUrl || "".equals(visitedUrl.trim())) { //抓取的地址爲空
visitedFrontier.putUrl(visitedCrawlUrl);
visitedCrawlUrl = (CrawlUrl)unvisitedFrontier.getNext();
continue;
}
try{
RetrievePage.downloadPage(visitedUrl); //下載頁面
Set<String> links = HtmlParserTool.extractLinks(visitedUrl, filter);
for(String link :links) {
if(!visitedFrontier.contains(link)
&&!unvisitedFrontier.contains(link) )
{
CrawlUrl unvisitedCrawlUrl = new CrawlUrl();
unvisitedCrawlUrl.setOriUrl(link);
unvisitedFrontier.putUrl(unvisitedCrawlUrl);
}
}
}catch(ConnectTimeoutException e) { //超時繼續讀下一個地址
visitedFrontier.putUrl(visitedCrawlUrl);
visitedCrawlUrl = (CrawlUrl)unvisitedFrontier.getNext();
num ++;
e.printStackTrace();
continue;
}catch(SocketTimeoutException e) {
visitedFrontier.putUrl(visitedCrawlUrl);
visitedCrawlUrl = (CrawlUrl)unvisitedFrontier.getNext();
num ++;
e.printStackTrace();
continue;
}
visitedCrawlUrl = (CrawlUrl)unvisitedFrontier.getNext();
num ++;
}while(BDBFrontier.threads >0 && num < 1000);
}
catch (IOException e) {
e.printStackTrace();
}
catch(Exception e) {
e.printStackTrace();
}
}
}（4）以Runnable接口形式實現d額多線程
……
public class MyCrawlerByThread extends MyCrawler implements Runnable{
private int threadId;
public MyCrawlerByThread(int id) {
this.threadId = id;
}
/**
* (non-Javadoc)
* @see java.lang.Runnable#run()
*/
@Override
public void run() {
try {
crawling(new String[]{CrawlConfig.CRAWL_PATH}, threadId);
}catch(Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
try {
long startTime=System.currentTimeMillis();
System.out.println("採集開始");
ArrayList<Thread> threadList = new ArrayList<Thread>(CrawlConfig.CRAWL_THREAD_NUM);
for(int i = 0 ; i < CrawlConfig.CRAWL_THREAD_NUM; i++) {
MyCrawlerByThread crawler = new MyCrawlerByThread(i);
Thread t = new Thread(crawler);
t.start();
threadList.add(t);
Thread.sleep(10L);
}
while(threadList.size() > 0) {
Thread child = (Thread) threadList.remove(0);
child.join();
}
System.out.println("採集結束");
long endTime=System.currentTimeMillis();
System.out.println("程序運行時間： "+(endTime-startTime)+"ms");
} catch(Exception e) {
e.printStackTrace();
}
}
}執行結果：
採集開始
……
採集結束
程序運行時間： 25777ms
最後對採集性能進行分析，先後採用LinkQueue隊列單線程、BerkeleyDB單線程、BerkeleyDB多線程方案進行網絡採集，測試數據對比如下：
綜上，多線程可以帶來明顯的性能提升。