Scrapy-Redis分佈式爬蟲組件

Scrapy-Redis介紹

Scrapy是一個框架，他本身是不支持分佈式的。如果我們想要做分佈式的爬蟲，就需要藉助一個組件叫做Scrapy-Redis，這個組件正是利用了Redis可以分佈式的功能，集成到Scrapy框架中，使得爬蟲可以進行分佈式。可以充分的利用資源（多個ip、更多帶寬、同步爬取）來提高爬蟲的爬行效率。
分佈式爬蟲的優點：可以充分利用多臺機器的帶寬；可以充分利用多臺機器的ip地址；多臺機器做，爬取效率更高。
分佈式爬蟲必須要解決的問題：分佈式爬蟲是好幾臺機器在同時運行，如何保證不同的機器爬取頁面的時候不會出現重複爬取的問題；同樣，分佈式爬蟲在不同的機器上運行，在把數據爬完後如何保證保存在同一個地方。
安裝：pip install scrapy-redis。

Scrapy-Redis架構圖：

從Redis獲取請求並去重；把爬下來的數據發送給Redis服務器。
Item Pipeline在接收到數據後發送給了Redis、Scheduler調度器調度數據也是從Redis中來的、並且其實數據去重也是在Redis中做的。

編寫Scrapy-Redis分佈式爬蟲：

要將一個Scrapy項目變成一個Scrapy-redis項目只需修改以下三點就可以了：

將爬蟲的類從scrapy.Spider變成scrapy_redis.spiders.RedisSpider；或者是從scrapy.CrawlSpider變成scrapy_redis.spiders.RedisCrawlSpider。
將爬蟲中的start_urls刪掉。增加一個redis_key=“xxx”。這個redis_key是爲了以後在redis中控制爬蟲啓動的。爬蟲的第一個url，就是在redis中通過這個發送出去的。
在配置文件中增加如下配置：

 # Scrapy-Redis相關配置
    # 確保request存儲到redis中
    SCHEDULER = "scrapy_redis.scheduler.Scheduler"

    # 確保所有爬蟲共享相同的去重指紋
    DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"

    # 設置redis爲item pipeline
    ITEM_PIPELINES = {
        'scrapy_redis.pipelines.RedisPipeline': 300
    }

    # 在redis中保持scrapy-redis用到的隊列，不會清理redis中的隊列，從而可以實現暫停和恢復的功能。
    SCHEDULER_PERSIST = True

    # 設置連接redis信息
    REDIS_HOST = '127.0.0.1'
    REDIS_PORT = 6379

運行爬蟲：
1. 在Redis服務器上。修改redis.conf的配置文件，將bind改成bind [自己的ip地址或者0.0.0.0]，其他機器才能訪問。
2. 在各個爬蟲服務器上安裝：python3-dev build-essential python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
3. 將寫好的爬蟲和需要安裝的包發送給各個爬蟲服務器；
4. 在爬蟲服務器上安裝包和運行爬蟲。進入爬蟲文件所在的路徑，然後輸入命令：scrapy runspider [爬蟲名字]
5. 在Redis服務器上，推入一個開始的url鏈接：redis-cli> lpush [redis_key] start_url開始爬取。

分佈式爬取房天下房源信息

fangspider.py

# -*- coding: utf-8 -*-
import re

import scrapy

from fang_redis.items import NewHouseItem

from fang_redis.items import ESFHouseItem
from scrapy_redis.spiders import RedisSpider

class FangspiderSpider(RedisSpider):
    name = 'fangspider'
    allowed_domains = ['fang.com']
    # start_urls = ['https://www.fang.com/SoufunFamily.htm']
    redis_key = "fang:start_url"

    def parse(self, response):
        trs = response.xpath("//div[@class='outCont']//tr")
        province = None
        for tr in trs:
            tds = tr.xpath(".//td[not(@class)]")
            province_text = tds[0]
            province_text = province_text.xpath(".//text()").get()
            province_text = re.sub(r'\s','',province_text)
            if province_text:
                province = province_text
            if province == "其它":
                continue
            city_id = tds[1]
            city_links = city_id.xpath(".//a")
            for city_link in city_links:
                city = city_link.xpath(".//text()").get()
                city_url = city_link.xpath(".//@href").get()
                url_module = city_url.split("//")
                scheme = url_module[0]
                domain = url_module[1]
                if "bj." in domain:
                    newhouse_url = "http://newhouse.fang.com/house/s/"
                    esf_url = 'http://esf.fang.com/'
                else:
                    newhouse_url = scheme + '//'+'newhouse.'+domain+"house/s/"
                    esf_url = scheme + "//" + "esf." + domain

                yield scrapy.Request(url=response.urljoin(newhouse_url),callback=self.parse_newhouse,meta={"info":(province,city)})

                yield scrapy.Request(url=response.urljoin(esf_url),callback=self.parse_esf,meta={"info":(province,city)})

    def parse_newhouse(self,response):
        province,city = response.meta.get("info")
        lis = response.xpath("//div[contains(@class,'nl_con')]/ul/li")
        for li in lis:
            name = li.xpath(".//div[@class='nlcd_name']/a/text()").get()
            if name:
                name = name.strip()
            house_type_list = li.xpath(".//div[contains(@class,'house_type')]/a/text()").getall()
            house_type_list = list(map(lambda x: re.sub(r"\s", "", x), house_type_list))
            rooms = list(filter(lambda x: x.endswith("居"), house_type_list))
            area = "".join(li.xpath(".//div[contains(@class,'house_type')]/text()").getall())
            area = re.sub(r"\s|－|/", "", area)
            address = li.xpath(".//div[@class='address']/a/@title").get()
            district_text = "".join(li.xpath(".//div[@class='address']/a//text()").getall())
            district = re.search(r".*\[(.+)\].*", district_text)
            if district:
                district = district.group(1)
            else:
                district = None
            sale = li.xpath(".//div[contains(@class,'fangyuan')]/span/text()").get()
            price = "".join(li.xpath(".//div[@class='nhouse_price']//text()").getall())
            price = re.sub(r"\s|廣告", "", price)
            origin_url = li.xpath(".//div[@class='nlcd_name']/a/@href").get()
            item = NewHouseItem(name=name, rooms=rooms, area=area, address=address, district=district, sale=sale,
                                price=price, origin_url=origin_url, province=province, city=city)
            yield item

        next_url = response.xpath("//div[@class='page']//a[@class='next']/@href").get()
        if next_url:
            yield scrapy.Request(url=response.urljoin(next_url), callback=self.parse_newhouse,
                                 meta={"info": (province, city)})

    def parse_esf(self,response):
        province, city = response.meta.get("info")
        dls = response.xpath("//div[@class='houseList']/dl")
        for dl in dls:
            item = ESFHouseItem(province=province, city=city)
            item['name'] = dl.xpath(".//p[@class='mt10']/a/span/text()").get()
            infos = dl.xpath(".//p[@class='mt12']/text()").getall()
            infos = list(map(lambda x: re.sub(r"\s", "", x), infos))
            for info in infos:
                if "廳" in info:
                    item['rooms'] = info
                elif '層' in info:
                    item['floor'] = info
                elif '向' in info:
                    item['toward'] = info
                else:
                    item['year'] = info.replace("建築年代：", "")
            item['address'] = dl.xpath(".//p[@class='mt10']/span/@title").get()
            item['area'] = dl.xpath(".//div[contains(@class,'area')]/p/text()").get()
            item['price'] = "".join(dl.xpath(".//div[@class='moreInfo']/p[1]//text()").getall())
            item['unit'] = "".join(dl.xpath(".//div[@class='moreInfo']/p[2]//text()").getall())
            detail_url = dl.xpath(".//p[@class='title']/a/@href").get()
            item['origin_url'] = response.urljoin(detail_url)
            yield item
        next_url = response.xpath("//a[@id='PageControl1_hlk_next']/@href").get()
        yield scrapy.Request(url=response.urljoin(next_url), callback=self.parse_esf, meta={"info": (province, city)})

middlewares.py

mport random
from scrapy import signals
class UserAgentDownloadMiddleware(object):
    USER_AGENTS=[
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2226.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 6.4; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36",
        "Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16",
        "Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14",
        "Mozilla/5.0 (Windows NT 6.0; rv:2.0) Gecko/20100101 Firefox/4.0 Opera 12.14",
        "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0) Opera 12.14",
        "Opera/12.80 (Windows NT 5.1; U; en) Presto/2.10.289 Version/12.02",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1",
        "Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0",
        "Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0",
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20130401 Firefox/31.0",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A",
        "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10",
        "Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko ) Version/5.1 Mobile/9B176 Safari/7534.48.3",
    ]

    def process_request(self,request,spider):
        user_agent = random.choice(self.USER_AGENTS)
        request.headers['User-Agent']=user_agent

items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class NewHouseItem(scrapy.Item):
    # 省份
    province = scrapy.Field()
    # 城市
    city  = scrapy.Field()
    # 小區的名字
    name = scrapy.Field()
    # 價格
    price = scrapy.Field()
    # 幾居。這個是個列表
    rooms = scrapy.Field()
    # 面積
    area = scrapy.Field()
    # 地址
    address = scrapy.Field()
    # 行政區
    district = scrapy.Field()
    # 是否在售
    sale = scrapy.Field()
    # 房天下詳情頁面的url
    origin_url = scrapy.Field()

class ESFHouseItem(scrapy.Item):
    # 省份
    province = scrapy.Field()
    # 城市
    city = scrapy.Field()
    # 小區的名字
    name = scrapy.Field()
    # 幾室幾廳
    rooms = scrapy.Field()
    # 層
    floor = scrapy.Field()
    # 朝向
    toward = scrapy.Field()
    # 年代
    year = scrapy.Field()
    # 地址
    address = scrapy.Field()
    # 建築面積
    area = scrapy.Field()
    # 總價
    price = scrapy.Field()
    # 單價
    unit = scrapy.Field()
    # 原始的url
    origin_url = scrapy.Field()

settings.py

# -*- coding: utf-8 -*-

# Scrapy settings for fang_redis project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'fang_redis'

SPIDER_MODULES = ['fang_redis.spiders']
NEWSPIDER_MODULE = 'fang_redis.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'fang_redis (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
  'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 '
                 '(KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36',
}

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'fang_redis.middlewares.FangRedisSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
   # 'fang_redis.middlewares.FangRedisDownloaderMiddleware': 543,
   'fang_redis.middlewares.UserAgentDownloadMiddleware': 300,
}

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
# ITEM_PIPELINES = {
#    'fang_redis.pipelines.FangPipeline': 300,
# }

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
# Scrapy-Redis相關配置
# 確保request存儲到redis中
SCHEDULER = "scrapy_redis.scheduler.Scheduler"

# 確保所有爬蟲共享相同的去重指紋
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"

# 設置redis爲item pipeline
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 300
}

# 在redis中保持scrapy-redis用到的隊列，不會清理redis中的隊列，從而可以實現暫停和恢復的功能。
SCHEDULER_PERSIST = True

# 設置連接redis信息
REDIS_HOST = '172.25.254.46'
REDIS_PORT = 6379

Scrapy-Redis分佈式爬蟲組件

Scrapy-Redis介紹

Scrapy-Redis架構圖：

編寫Scrapy-Redis分佈式爬蟲：

分佈式爬取房天下房源信息

【簡寫Mybatis-02】註冊機的實現以及SqlSession處理

手繪二維碼

.NET藉助虛擬網卡實現一個簡單異地組網工具

爬蟲urllib庫及requests模塊

CrawlSpider模板

Scrapy-Redis分佈式爬蟲組件

scrapy的下載器中間件及配置文件

Scrapy Shell 和 Request、Response對象

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結