Scrapy的指紋去重原理

指紋去重算法代碼在scrapy.utils.request

def request_fingerprint(request, include_headers=None):
    if include_headers:
        include_headers = tuple(to_bytes(h.lower())
                                 for h in sorted(include_headers))
    cache = _fingerprint_cache.setdefault(request, {})
    if include_headers not in cache:
        fp = hashlib.sha1()
        fp.update(to_bytes(request.method))
        fp.update(to_bytes(canonicalize_url(request.url)))
        fp.update(request.body or b'')
        if include_headers:
            for hdr in include_headers:
                if hdr in request.headers:
                    fp.update(hdr)
                    for v in request.headers.getlist(hdr):
                        fp.update(v)
        cache[include_headers] = fp.hexdigest()
    return cache[include_headers]

指紋去重前提

request.dont_filter=False

指紋由何生成？

請求方法：request.method

請求地址：request.url

請求體：request.body

簡而言之，根據請求方法、請求地址、請求體生成指紋，若指紋已存在則過濾，否則加入請求隊列。