運行環境
PyCharm 2018.1
Scrapy 1.5.1
Python 3.5.4
Windows 10
Scrapy爬蟲已經編寫完成,運行(scrapy crawl <spider_name> ) 發生如下錯誤
Traceback (most recent call last):
...
File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\cmdline.py", line 149, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\crawler.py", line 249, in __init__
super(CrawlerProcess, self).__init__(settings)
File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\crawler.py", line 137, in __init__
self.spider_loader = _get_spider_loader(settings)
File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\crawler.py", line 336, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\settings\__init__.py", line 372, in frozencopy
copy = self.copy()
File "d:\pycharmprojects\commercialcrawler\venv\lib\site-packages\scrapy\settings\__init__.py", line 354, in copy
return copy.deepcopy(self)
...
File "D:\Python3.5\lib\copy.py", line 166, in deepcopy
y = copier(memo)
TypeError: cannot deepcopy this pattern object
查閱網上發現並沒有出現和我類似的錯誤,於是從頭開始檢查traceback。
發現scrapy.crawler._get_spider_loader()
裏調用了settings.frozencopy()
方法,
找到frozencopy()
:
import copy
def frozencopy(self):
"""
Return an immutable copy of the current settings.
Alias for a :meth:`~freeze` call in the object returned by :meth:`copy`.
"""
copy = self.copy()
copy.freeze()
return copy
def copy(self):
"""
Make a deep copy of current settings.
This method returns a new instance of the :class:`Settings` class,
populated with the same values and their priorities.
Modifications to the new object won't be reflected on the original
settings.
"""
return copy.deepcopy(self)
def freeze(self):
"""
Disable further changes to the current settings.
After calling this method, the present state of the settings will become
immutable. Trying to change values through the :meth:`~set` method and
its variants won't be possible and will be alerted.
"""
self.frozen = True
可見此方法‘’凍結‘’了現有的settings
,使用了copy.deepcopy()
深拷貝。
此時再在網上搜索深拷貝類似的錯誤,並且檢查settings.py裏面是否有無法深拷貝的對象。
發現我在settings.py裏面調用了configparser,並且以變量名CONFIG
實例化了一個ConfigParser對象
import configparser
CONFIG= configparser.ConfigParser()
由於scrapy 只會讀取變量名爲大寫的變量,將CONFIG
更名爲config
。
Problem solved!