解決ValueError('Missing scheme in request url: %s' % self._url)

原創 LCYong_ 2020-02-20 14:26

使用scrapy的ImagesPipeline爬取圖片的時候，運行報錯

Traceback (most recent call last):
  File "/home/lcy/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/lcy/.local/lib/python2.7/site-packages/scrapy/pipelines/media.py", line 62, in process_item
    requests = arg_to_iter(self.get_media_requests(item, info))
  File "/home/lcy/.local/lib/python2.7/site-packages/scrapy/pipelines/images.py", line 147, in get_media_requests
    return [Request(x) for x in item.get(self.images_urls_field, [])]
  File "/home/lcy/.local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/home/lcy/.local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 57, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: h

查找了相關的文檔，瞭解到使用ImagesPipeline傳入的url地址必須是一個list，在傳入一個list的時候pipeline處理的速度要快得多，而我寫的是一個字符串，所以報錯，所以我們需要修改一下傳入的url格式就行了



源碼附上：

修改前：

# -*- coding: utf-8 -*-
import scrapy
from imgspider.items import QiubaiPicItem
import sys
reload(sys)
sys.setdefaultencoding( "utf-8" )
class QiubaipicSpider(scrapy.Spider):
    name = "qiubaiPic"
    allowed_domains = ["qiushibaike.com"]
    start_urls = ['http://qiushibaike.com/']

    def parse(self, response):
        # page_value=response.xpath('//*[@id="content-left"]/ul/li[8]/a/span/text()').extract()[0]
        # for page in range(1,int(page_value)):
        #     url='http://www.qiushibaike.com/pic/page/'+str(page)
        #     yield scrapy.Request(url,callback=self.parse_detail)

        url='http://www.qiushibaike.com/pic/page/3'
        yield scrapy.Request(url,callback=self.parse_detail)

    def parse_detail(self,response):
        item=[]  
        divs=response.xpath('//*[@id="content-left"]/div[@class="article block untagged mb15"]')
        for div in divs:
            QiubaiPic=QiubaiPicItem()
            src=div.xpath('div[@class="thumb"]/a/img/@src').extract()[0]
            img_path='http://'+src[2:]   
            QiubaiPic['img']=img_path
            item.append(QiubaiPic)
        return item





修改後：

# -*- coding: utf-8 -*-
import scrapy
from imgspider.items import QiubaiPicItem
import sys
reload(sys)
sys.setdefaultencoding( "utf-8" )
class QiubaipicSpider(scrapy.Spider):
    name = "qiubaiPic"
    allowed_domains = ["qiushibaike.com"]
    start_urls = ['http://qiushibaike.com/']

    def parse(self, response):
        # page_value=response.xpath('//*[@id="content-left"]/ul/li[8]/a/span/text()').extract()[0]
        # for page in range(1,int(page_value)):
        #     url='http://www.qiushibaike.com/pic/page/'+str(page)
        #     yield scrapy.Request(url,callback=self.parse_detail)

        url='http://www.qiushibaike.com/pic/page/3'
        yield scrapy.Request(url,callback=self.parse_detail)

    def parse_detail(self,response):
        item=[]
        img_paths=[]
        divs=response.xpath('//*[@id="content-left"]/div[@class="article block untagged mb15"]')
        for div in divs:
            QiubaiPic=QiubaiPicItem()
            src=div.xpath('div[@class="thumb"]/a/img/@src').extract()[0]
            img_path='http://'+src[2:]
            img_paths.append(img_path)
        QiubaiPic['img']=img_paths
        item.append(QiubaiPic)
        return item


setting.py文件 

# -*- coding: utf-8 -*-

import random

BOT_NAME = 'imgspider'

SPIDER_MODULES = ['imgspider.spiders']
NEWSPIDER_MODULE = 'imgspider.spiders'
#瀏覽器請求頭，這個必須要有
USER_AGENT_LIST=[
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
    "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1" \
    "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11", \
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6", \
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6", \
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1", \
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5", \
    "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5", \
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", \
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", \
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", \
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", \
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", \
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", \
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", \
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", \
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3", \
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24", \
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"

]
ua= random.choice(USER_AGENT_LIST)
if ua:
    USER_AGENT =ua
    print ua
else:
    USER_AGENT="Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"

#是否遵循robots協定
ROBOTSTXT_OBEY = False
#線程數量
CONCURRENT_REQUESTS = 32
#下載延遲單位秒
DOWNLOAD_DELAY = 3
#cookies開關，建議禁用
COOKIES_ENABLED = False

# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    'scrapy.pipelines.images.ImagesPipline':1}
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}

IMAGES_URLS_FIELD = 'img'
IMAGES_STORE = r'/home/lcy/pics'
LOG_FILE="scrapy.log"





























LCYong_



發佈了100 篇原創文章 · 獲贊 50 · 訪問量 29萬+





私信

關注

















發表評論














所有評論



還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.







相關文章








Requests庫基礎入門






1.Requests庫入門
中文文檔：https://requests.readthedocs.io/zh_CN/latest/
gittub地址：https://github.com/requests/requests
更多信息






 Pang_ling

2020-07-06 08:31:07









爬蟲學習：scrapy相關的知識






Scrapy 是開源的爬蟲框架，快速強大，只需要寫少量的代碼即可完成爬取任務，容易擴展，添加新的功能模塊
1. scrapy 在Mac上的安裝
可以使用pip安裝Scrapy
運行命令 sudo pip ins






 lovive

2020-07-03 11:57:40









Python爬蟲庫學習筆記-requests






安裝
（1）利用pip安裝，命令行直接輸入pip install requests
（2）利用easy_install安裝，命令行直接輸入easy_install requests
基本請求
（1）requests庫提供h






 huangjx36

2020-06-29 04:02:16









Python爬蟲實踐筆記（三）






網頁式登錄的無線網絡重連–本質
註釋：事實上這個程序就是在連接了某個無線網之後，登錄該無線網指定的頁面，然後就可以正常上網了。跟爬蟲的連接和登錄網頁的本質是一樣的，只是需要分析一下請求和響應數據，以至於能夠實現重連。
第一步：分






 huangjx36

2020-06-29 04:02:16









Python爬蟲實踐筆記（二）






無登錄百度貼吧的帖子
基本步驟：
①分析url ②獲取頁面 ③提取信息 ④文本處理 ⑤寫入文件
備註：
基本框架跟筆記（一）很相似，只是多了很多細節需要處理，所以這裏着重描述細節的處理。
第一步：分析url
（1）像段子、貼吧






 huangjx36

2020-06-29 04:02:16









Python爬蟲學習筆記--MySQLdb模塊






mysql數據庫和MySQLdb模塊相關問題
下載與安裝
（1）首先，python要使用數據庫不是僅僅安裝相應的模塊就行的，安裝的模塊，例如這裏的MySQLdb模塊只是一個訪問mysql數據庫的接口。所以還是必須要下載一個數據庫的






 huangjx36

2020-06-29 04:02:06









爬蟲實戰——xpath爬取電影天堂






from lxml import etree
import requests
HEADERS = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Apple






 SLEEPYHEAD's Blog

2020-06-28 05:25:00









爬蟲實戰——古詩文網古詩爬取






import re
import requests
# url = 'https://www.gushiwen.org/default_1.aspx'
def parse_page(url):
headers = {






 SLEEPYHEAD's Blog

2020-06-28 05:25:00









查看網頁編碼方式的通用方法






在python爬蟲等各種情景模式下，往往需要查看網頁的編碼方式。下面是通用，簡單的方法。
在各種瀏覽器打開的任意頁面上使用F12功能鍵，即可使用開發者工具，在窗口console標籤下，鍵入 "document.charset" 即可查看網






 张少锋的博客

2020-06-17 08:30:54









python爬蟲學習筆記 4.5 （Spider）






 还算小萌新？

2020-05-10 08:53:53









python爬蟲學習筆記 4.7 （Request/Response）






 还算小萌新？

2020-05-10 08:53:53









python爬蟲學習筆記 4.9 （settings）






 还算小萌新？

2020-05-10 08:53:53









python爬蟲學習筆記 4.6 （CrawlSpider）






 还算小萌新？

2020-05-10 08:53:53









python爬蟲學習筆記 4.4 （Item Pipline）






 还算小萌新？

2020-05-10 08:53:53









python爬蟲學習筆記 4.8 （Downloader Middlewares）






 还算小萌新？

2020-05-10 08:53:53