phantomjs安裝（參考http://www.cnblogs.com/yestreenstars/p/5511212.html）

# 安裝依賴軟件
yum -y install wget fontconfig

# 下載PhantomJS
wget -P /tmp/ https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-i686.tar.bz2

# 解壓
tar xjf /tmp/phantomjs-2.1.1-linux-i686.tar.bz2 -C /usr/local/

# 重命名
mv /usr/local/phantomjs-2.1.1-linux-i686 /usr/local/phantomjs

# 建立軟鏈接
ln -s /usr/local/phantomjs/bin/phantomjs /usr/bin/

selenuim安裝參考官網教程，有時需要裝瀏覽器驅動

這次目標網頁爲異步的，內容爲JS生成的部分，JS中還帶有翻頁，每一頁的內容都要爬下來。

使用selenuim+phantomjs模擬瀏覽器操作，phantomjs爲不帶窗口的瀏覽器，

這裏需要注意執行完後webdriver需要退出，否則進程會一直保留，退出方法爲quit()，詳細用法參考代碼。

代碼如下：

# -*- coding: UTF-8 -*-
import re
import time  
import Queue  
import urllib2  
import threading 
import pickle
from bs4 import BeautifulSoup  
from selenium import webdriver  
from selenium import * 

def get_source(page_num):
    list = []
    #加載瀏覽器驅動
    browser = webdriver.PhantomJS(executable_path="/usr/local/phantomjs/bin/phantomjs")
    #窗口最大化
    browser.maximize_window()  
    #與網址建立連接
    browser.get('------------url------------')
    #找到某元素並點擊
    browser.find_element_by_xpath("//a[contains(text(),'年報') and @data-toggle='tab']").click()
    #等待5秒
    time.sleep(5)
    #將源碼賦給beautifulsoup去解析
    soup = BeautifulSoup(browser.page_source)
    parser(soup)
    list.extend(parser(soup))
    pre = "//a[@class='classStr' and @id='idStr' and @page='"
    suf = "' and @name='nameStr' and @href='javascript:;']"
    for i in range(2,page_num):
        browser.find_element_by_xpath(pre + str(i) + suf).click()
        time.sleep(10)
        soup = BeautifulSoup(browser.page_source)
        list.extend(parser(soup))

    browser.quit()

    return list


def parser(soup):
    pdf_path_list = []
    domain = "------url---------"
    #用正則找到所有‘href’屬性包含‘pdf’的標籤
    for tag in soup.find_all(href=re.compile("pdf")):
        hre =  tag.get('href')
        if hre.find("main_cn") < 0:
            pdf_path = domain + hre
            pdf_path_list.append(pdf_path)
            print pdf_path
    return pdf_path_list        


def main():
    list = get_source(96)
    f = open('pdf_url.txt', 'w')
    pickle.dump(list, f)

main()

selenuim元素定位參考  http://www.cnblogs.com/yufeihlf/p/5717291.html#test2  

和  http://www.cnblogs.com/yufeihlf/p/5764807.html

BeautifulSoup解析頁面參考 http://cuiqingcai.com/1319.html

以上代碼得到了要下載的pdf的url，接着使用requests下載，代碼如下：

import requests
import time
import pickle

def download_file(url):  
    # NOTE the stream=True parameter 
    local_filename = "./pdfs/" + url[url.rfind('/') + 1:]
    print local_filename
    r = requests.get(url, stream=True) 
    if r.status_code == 200:
        with open(local_filename, 'wb') as f:  
            for chunk in r.iter_content(chunk_size=1024):  
                if chunk: # filter out keep-alive new chunks  
                    f.write(chunk)  
                    f.flush()  
        print(url + " has been downloaded")
    else: 
        print("r.status_code = " + str(r.status_code) + ", download failed")

f = open("pdf_url.txt", "r")
for line in f.readlines():
    download_file(line.strip())
    time.sleep(10)