Chrome 瀏覽器插件 Web Scraper 可輕鬆實現網頁數據的爬取，還不用考慮爬蟲中的登陸、驗證碼、異步加載等複雜問題。
先貼上爬蟲58上爬數據的sitemap如下：
{"_id":"hefeitongcheng","startUrl":["https://hf.58.com/shushanqu/baihuochaoshi/s32/?PGTID=0d306b32-0034-8449-027b-ed96503664b1&ClickID=1"],"selectors":[{"id":"click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".list-main-style li","multiple":true,"delay":"5000","clickElementSelector":"strong span","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"link","type":"SelectorLink","parentSelectors":["click"],"selector":".title a","multiple":false,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["link"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"jiage","type":"SelectorText","parentSelectors":["link"],"selector":".house_basic_title_money span","multiple":false,"regex":"","delay":0},{"id":"add","type":"SelectorText","parentSelectors":["link"],"selector":"p.p_2","multiple":false,"regex":"","delay":0}]}
Web Scraper 抓取流程及要點：
安裝Web Scraper插件後，三步完成爬取操作
1、Create new sitemap（創建爬取項目）
2、選取爬取網頁中的內容，點~點~點，操作
3、開啓爬取，下載CSV數據
其中最關鍵的是第二步，兩個要點：
- 先選中數據塊 Element，每塊數據我們在頁面上取，都是重複的，選中 Multiple
- 在數據塊中再取需要的數據字段(上圖Excel中的列)
爬取大量數據的要點，在於掌握分頁的控制。
分頁分爲3種情況：
1. URL 參數分頁(比較規整方式) ?page=2 或 ?page=[1-27388]
2. 滾動加載，點擊“加載更多” 加載頁面數據 Element scroll down
3. 點擊分頁數字標籤(包括“下一頁”標籤) Link 或 Element click
其它範例A：jd上爬hw p30價格信息
{"_id":"huaweip30","startUrl":["https://search.jd.com/Search?keyword=%E5%8D%8E%E4%B8%BAp30%20512&enc=utf-8&wq=%E5%8D%8E%E4%B8%BAp30%20512&pvid=ed449bf16e44461fac90ff6fae2e66cd"],"selectors":[{"id":"element","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.gl-i-wrap","multiple":true,"delay":"1500","clickElementSelector":".p-num a:nth-of-type(3)","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"name","type":"SelectorText","parentSelectors":["element"],"selector":"a em","multiple":false,"regex":"","delay":0},{"id":"jiage","type":"SelectorText","parentSelectors":["element"],"selector":"div.p-price","multiple":false,"regex":"","delay":0}]}
其它範例B：baidu上爬關鍵字信息
{"_id":"wailaizhu","startUrl":["https://www.baidu.com/s?wd=wailaizhu%20h0101&pn=0&oq=wailaizhu%20h0101&tn=baiduhome_pg&ie=utf-8&rsv_idx=2&rsv_pq=f62d151f0001156d&rsv_t=5b15EoMWRlm3%2BeroyWXBKI%2FDZ3H0BlGKJ6lNa6mmYBo4nNDUeJNeeN8BvgiE9S9Orivd"],"selectors":[{"id":"element","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#content_left","multiple":true,"delay":"1500","clickElementSelector":"a span.pc","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"name","type":"SelectorText","parentSelectors":["element"],"selector":"a","multiple":false,"regex":"","delay":0},{"id":"body","type":"SelectorText","parentSelectors":["element"],"selector":"_parent_","multiple":false,"regex":"","delay":0}]}