六、Scrapy（一）| 入門案例及本地運行測試

原創 一把小强 2019-10-27 01:15

1、案例

創建 test.py 文件

# encoding: utf-8

"""
@author: sunxianpeng
@file: test.py
@time: 2019/10/26 18:37
"""
import scrapy

class Main(scrapy.Spider):
    def __init__(self):
        pass
    # 爬蟲的名字
    name ="quote"
    # 起始的url列表
    start_urls = ['http://quotes.toscrape.com/']

    # 固定函數，固定寫法
    def parse(self, response):
        # css和 xpath 都可以實現
        # quotes = response.css("div.quote")
        quotes = response.xpath('//div[@class="quote"]')#和上述作用一樣
        for quote in quotes:
            yield {
                # extrack_first提取第一個選擇器的內容
                "text": quote.css('span.text::text').extract_first(),
                "author": quote.xpath('./span/small/text').extract_first()
            }
        next_page = response.xpath('//li[@class="next"]/a/@href').extract_first()

        if next_page:
            # page下載完後，交給誰處理(再交給parse函數處理)
            yield response.follow(next_page,self.parse)



if __name__ == '__main__':
    m = Main()

2、本地運行測試

（1）Pycharm下，點擊下方按鈕

（2）切換到 .py文件所在的路徑

cd spider/scrapy/

（3）運行命令

scrapy runspider test.py

（4）查看控制檯信息

2019-10-26 19:35:45 [scrapy.utils.log] INFO: Scrapy 1.7.4 started (bot: scrapybot)
2019-10-26 19:35:45 [scrapy.utils.log] INFO: Versions: lxml 4.3.3.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.2.0, Python 3.
7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1c  28 May 2019), cryptography 2.7, Platform Windows-7-6.1
.7601-SP1
2019-10-26 19:35:45 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2019-10-26 19:35:45 [scrapy.extensions.telnet] INFO: Telnet Password: 3566c311a724af29
2019-10-26 19:35:45 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2019-10-26 19:35:45 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-10-26 19:35:45 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-10-26 19:35:45 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-10-26 19:35:45 [scrapy.core.engine] INFO: Spider opened
2019-10-26 19:35:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-10-26 19:35:45 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-10-26 19:35:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/> (referer: None)
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”', 'author'
: None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”", 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“Try not to become a man of success. Rather become a man of value.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“It is better to be hated for what you are than to be loved for what you are not.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': "“I have not failed. I've just found 10,000 ways that won't work.”", 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”", 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
{'text': '“A day without sunshine is like, you know, night.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: http://quotes.toscrape.com/)
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': "“This life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to deci
de how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with
 you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers,
well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't gi
ve up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because
you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, t
hen who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much
 to smile about.”", 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': '“It takes a great deal of bravery to stand up to our enemies, but just as much to stand up to our friends.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': "“If you can't explain it to a six year old, you don't understand it yourself.”", 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': "“You may not be her first, her last, or her only. She loved before she may love again. But if she loves you now, what else matters? She's not p
erfect—you aren't either, and the two of you may never be perfect together but if she can make you laugh, cause you to think twice, and admit to being hu
man and making mistakes, hold onto her and give her the most you can. She may not be thinking about you every second of the day, but she will give you a p
art of her that she knows you can break—her heart. So don't hurt her, don't change her, don't analyze and don't expect more than she can give. Smile when
 she makes you happy, let her know when she makes you mad, and miss her when she's not there.”", 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': '“I like nonsense, it wakes up the brain cells. Fantasy is a necessary ingredient in living.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': '“I may not have gone where I intended to go, but I think I have ended up where I needed to be.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': "“The opposite of love is not hate, it's indifference. The opposite of art is not ugliness, it's indifference. The opposite of faith is not here
sy, it's indifference. And the opposite of life is not death, it's indifference.”", 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': '“It is not a lack of love, but a lack of friendship that makes unhappy marriages.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': '“Good friends, good books, and a sleepy conscience: this is the ideal life.”', 'author': None}
2019-10-26 19:35:46 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
{'text': '“Life is what happens to us while we are making other plans.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/3/> (referer: http://quotes.toscrape.com/page/2/)
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“I love you without knowing how, or when, or from where. I love you simply, without problems or pride: I love you in this way because I do not
know any other way of loving but this, in which there is no I or you, so intimate that your hand upon my chest is my hand, so intimate that when I fall as
leep your eyes close.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“For every minute you are angry you lose sixty seconds of happiness.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“If you judge people, you have no time to love them.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“Anyone who thinks sitting in church can make you a Christian must also think that sitting in a garage can make you a car.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“Beauty is in the eye of the beholder and it may be necessary from time to time to give a stupid or misinformed beholder a black eye.”', 'auth
or': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“Today you are You, that is truer than true. There is no one alive who is Youer than You.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“If you want your children to be intelligent, read them fairy tales. If you want them to be more intelligent, read them more fairy tales.”', '
author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“It is impossible to live without failing at something, unless you live so cautiously that you might as well not have lived at all - in which c
ase, you fail by default.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“Logic will get you from A to Z; imagination will get you everywhere.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/3/>
{'text': '“One good thing about music, when it hits you, you feel no pain.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/4/> (referer: http://quotes.toscrape.com/page/3/)
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': "“The more that you read, the more things you will know. The more that you learn, the more places you'll go.”", 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': '“Of course it is happening inside your head, Harry, but why on earth should that mean that it is not real?”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': '“The truth is, everyone is going to hurt you. You just got to find the ones worth suffering for.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': '“Not all of us can do great things. But we can do small things with great love.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': '“To the well-organized mind, death is but the next great adventure.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': "“All you need is love. But a little chocolate now and then doesn't hurt.”", 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': "“We read to know we're not alone.”", 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': '“Any fool can know. The point is to understand.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': '“I have always imagined that Paradise will be a kind of library.”', 'author': None}
2019-10-26 19:35:47 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/4/>
{'text': '“It is never too late to be what you might have been.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/5/> (referer: http://quotes.toscrape.com/page/4/)
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“A reader lives a thousand lives before he dies, said Jojen. The man who never reads lives only one.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“You can never get a cup of tea large enough or a book long enough to suit me.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“You believe lies so you eventually learn to trust no one but yourself.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“If you can make a woman laugh, you can make her do anything.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“Life is like riding a bicycle. To keep your balance, you must keep moving.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“The real lover is the man who can thrill you by kissing your forehead or smiling into your eyes or just staring into space.”', 'author': None
}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': "“A wise girl kisses but doesn't love, listens but doesn't believe, and leaves before she is left.”", 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“Only in the darkness can you see the stars.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“It matters not what someone is born, but what they grow to be.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/5/>
{'text': '“Love does not begin and end the way we seem to think it does. Love is a battle, love is a war; love is a growing up.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/6/> (referer: http://quotes.toscrape.com/page/5/)
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“There is nothing I would not do for those who are really my friends. I have no notion of loving people by halves, it is not my nature.”', 'au
thor': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“Do one thing every day that scares you.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“I am good, but not an angel. I do sin, but I am not the devil. I am just a small girl in a big world trying to find someone to love.”', 'auth
or': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“If I were not a physicist, I would probably be a musician. I often think in music. I live my daydreams in music. I see my life in terms of mus
ic.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“If you only read the books that everyone else is reading, you can only think what everyone else is thinking.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“The difference between genius and stupidity is: genius has its limits.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': "“He's like a drug for you, Bella.”", 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“There is no friend as loyal as a book.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': '“When one door of happiness closes, another opens; but often we look so long at the closed door that we do not see the one which has been opene
d for us.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/6/>
{'text': "“Life isn't about finding yourself. Life is about creating yourself.”", 'author': None}
2019-10-26 19:35:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/7/> (referer: http://quotes.toscrape.com/page/6/)
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': "“That's the problem with drinking, I thought, as I poured myself a drink. If something bad happens you drink in an attempt to forget; if someth
ing good happens you drink in order to celebrate; and if nothing happens you drink to make something happen.”", 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': '“You don’t forget the face of the person who was your last hope.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': "“Remember, we're madly in love, so it's all right to kiss me anytime you feel like it.”", 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': '“To love at all is to be vulnerable. Love anything and your heart will be wrung and possibly broken. If you want to make sure of keeping it int
act you must give it to no one, not even an animal. Wrap it carefully round with hobbies and little luxuries; avoid all entanglements. Lock it up safe in
the casket or coffin of your selfishness. But in that casket, safe, dark, motionless, airless, it will change. It will not be broken; it will become unbre
akable, impenetrable, irredeemable. To love is to be vulnerable.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': '“Not all those who wander are lost.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': '“Do not pity the dead, Harry. Pity the living, and, above all those who live without love.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': '“There is nothing to writing. All you do is sit down at a typewriter and bleed.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': '“Finish each day and be done with it. You have done what you could. Some blunders and absurdities no doubt crept in; forget them as soon as you
 can. Tomorrow is a new day. You shall begin it serenely and with too high a spirit to be encumbered with your old nonsense.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': '“I have never let my schooling interfere with my education.”', 'author': None}
2019-10-26 19:35:48 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/7/>
{'text': "“I have heard there are troubles of more than one kind. Some come from ahead and some come from behind. But I've bought a big bat. I'm all read
y you see. Now my troubles are going to have troubles with me!”", 'author': None}
2019-10-26 19:35:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/8/> (referer: http://quotes.toscrape.com/page/7/)
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': '“If I had a flower for every time I thought of you...I could walk through my garden forever.”', 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': '“Some people never go crazy. What truly horrible lives they must lead.”', 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': '“The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.”', 'author': None
}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': '“Think left and think right and think low and think high. Oh, the thinks you can think up if only you try!”', 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': "“What really knocks me out is a book that, when you're all done reading it, you wish the author that wrote it was a terrific friend of yours an
d you could call him up on the phone whenever you felt like it. That doesn't happen much, though.”", 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': '“The reason I talk to myself is because I’m the only one whose answers I accept.”', 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': "“You may say I'm a dreamer, but I'm not the only one. I hope someday you'll join us. And the world will live as one.”", 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': '“I am free of all prejudice. I hate everyone equally. ”', 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': "“The question isn't who is going to let me; it's who is going to stop me.”", 'author': None}
2019-10-26 19:35:50 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/8/>
{'text': "“′Classic′ - a book which people praise and don't read.”", 'author': None}
2019-10-26 19:35:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/9/> (referer: http://quotes.toscrape.com/page/8/)
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“Anyone who has never made a mistake has never tried anything new.”', 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': "“A lady's imagination is very rapid; it jumps from admiration to love, from love to matrimony in a moment.”", 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“Remember, if the time should come when you have to make a choice between what is right and what is easy, remember what happened to a boy who w
as good, and kind, and brave, because he strayed across the path of Lord Voldemort. Remember Cedric Diggory.”', 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“I declare after all there is no enjoyment like reading! How much sooner one tires of any thing than of a book! -- When I have a house of my ow
n, I shall be miserable if I have not an excellent library.”', 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“There are few people whom I really love, and still fewer of whom I think well. The more I see of the world, the more am I dissatisfied with it
; and every day confirms my belief of the inconsistency of all human characters, and of the little dependence that can be placed on the appearance of meri
t or sense.”', 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“Some day you will be old enough to start reading fairy tales again.”', 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“We are not necessarily doubting that God will do the best for us; we are wondering how painful the best will turn out to be.”', 'author': Non
e}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“The fear of death follows from the fear of life. A man who lives fully is prepared to die at any time.”', 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“A lie can travel half way around the world while the truth is putting on its shoes.”', 'author': None}
2019-10-26 19:35:51 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/9/>
{'text': '“I believe in Christianity as I believe that the sun has risen: not only because I see it, but because by it I see everything else.”', 'author
': None}
2019-10-26 19:35:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/10/> (referer: http://quotes.toscrape.com/page/9/)
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“The truth." Dumbledore sighed. "It is a beautiful and terrible thing, and should therefore be treated with great caution.”', 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': "“I'm the one that's got to die when it's time for me to die, so let me live my life the way I want to.”", 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“To die will be an awfully big adventure.”', 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“It takes courage to grow up and become who you really are.”', 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“But better to get hurt by the truth than comforted with a lie.”', 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“You never really understand a person until you consider things from his point of view... Until you climb inside of his skin and walk around in
 it.”', 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“You have to write the book that wants to be written. And if the book will be too difficult for grown-ups, then you write it for children.”',
'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“Never tell the truth to people who are not worthy of it.”', 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': "“A person's a person, no matter how small.”", 'author': None}
2019-10-26 19:35:52 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/10/>
{'text': '“... a mind needs books as a sword needs a whetstone, if it is to keep its edge.”', 'author': None}
2019-10-26 19:35:52 [scrapy.core.engine] INFO: Closing spider (finished)
2019-10-26 19:35:52 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2642,
 'downloader/request_count': 10,
 'downloader/request_method_count/GET': 10,
 'downloader/response_bytes': 24444,
 'downloader/response_count': 10,
 'downloader/response_status_count/200': 10,
 'elapsed_time_seconds': 6.284,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 10, 26, 11, 35, 52, 60200),
 'item_scraped_count': 100,
 'log_count/DEBUG': 110,
 'log_count/INFO': 10,
 'request_depth_max': 9,
 'response_received_count': 10,
 'scheduler/dequeued': 10,
 'scheduler/dequeued/memory': 10,
 'scheduler/enqueued': 10,
 'scheduler/enqueued/memory': 10,
 'start_time': datetime.datetime(2019, 10, 26, 11, 35, 45, 776200)}
2019-10-26 19:35:52 [scrapy.core.engine] INFO: Spider closed (finished)

 

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

【Python爬蟲】基本原理和框架

開發者工具介紹 參考：https://blog.csdn.net/m0_37724356/article/details/79884006 右擊網頁-檢查，或者F12，打開網頁開發者工具 get方式將請求的參數包含在url裏面

Mercy92 2020-07-08 10:40:50

【Python 爬蟲】使用友盟API獲取數據

一、需求 每天需要從友盟網站獲取若干應用如下信息 二、實現-分解 1）獲取api祕鑰 #獲取api祕鑰 def authorize(user, pasw): url = 'http://api.umeng.com/aut

Mercy92 2020-07-08 10:40:48

【Python 爬蟲】使用新榜API獲取數據

一、需求 每日獲取新榜網站收藏的微信公衆號對應的【點贊數、排名、頭條閱讀數】 二、實現-分解 1）獲取新榜api 打開並登錄新榜網站-數據服務-數據API 點擊試用即可，到達控制檯，你的賬號就獲得API密鑰，以及2000unit額

Mercy92 2020-07-08 10:40:48

Python爬蟲之selenium爬取英雄聯盟官網英雄皮膚圖片下載到本地和保存到數據庫

從英雄聯盟皮膚網站的網頁源代碼中獲取不到英雄的皮膚地址 通過selenium可以輕鬆獲取想要的內容 源碼展示 from selenium import webdriver from time import sleep from

xyl180808 2020-07-08 00:23:55

python爬蟲：番號挖掘機！可下載直接運行。

最近一直在學習python，想寫一些練習。看別人都是爬壁紙什麼的，感覺那些也沒啥意思，所有我就去爬番號了。 如果不想自己複製代碼可以自行下載 點我下載，提取碼：9nas 爬蟲目錄 __pycache__:這個文件夾python

你就叫我李大帅 2020-07-07 21:45:58

使用Beautifulsoup解析網頁遇到的問題

今天遇到一個網頁，按往常的老辦法 soup = BeautifulSoup(content, 'lxml') 打印soup發現少了很多數據，剛開始還以爲反爬，後來經過驗證不是。、 那麼就是解析問題，我換成了 soup = Bea

李孟笛 2020-07-07 18:21:30

python如何下載MP4視頻和獲取時長

我們做爬蟲工作時，經常會遇到要把爬取的視頻下載到本地做存儲，那麼我們如何使用python去下載視頻呢，下載視頻不是本文章的重點，在此廢話不多說，直接上完整的代碼： def download_file(url, base_path, fi

公众号菜鸟童靴 2020-07-07 11:53:23

execjs執行js代碼報錯:Exception in thread Thread-1

最近在爬一個js數據加密的網站的時候,出了點問題,困擾了我兩天 直接運行js文件的時候正常,但是用execjs運行js代碼的時候總是會報錯    最後翻了很多博客之後,終於找到了原因:原因是有一個程序在使用TextIOWrapper 類

公众号菜鸟童靴 2020-07-07 11:53:23

Python爬蟲精簡步驟 HTML基礎（上）

開門見山，HTML的學習可分爲三個層次。 讀懂，修改，編寫。 讀懂：只有讀懂了HTML，才能看得懂網頁結構，纔有可能運用Python的其他模塊去解析數據和提取數據。想寫爬蟲程序一定要先學好HTML基礎。 修改：在讀懂HTML文檔的

Lora鳃鳃 2020-07-07 11:50:32

一個簡單的校園網登錄程序 || 爬蟲+tkinter

僅用於登錄中國礦業大學校園網 可以自動記錄用戶名和密碼,省去二次填寫的麻煩 效果 代碼 import tkinter as tk import requests import re import tkinte

阿腾木 2020-07-07 06:57:17

python爬蟲初學的小記錄

self.headers = {“User-Agent”: “Mozilla/5.0”} 對headers的解釋： User Agent中文名爲用戶代理，是Http協議中的一部分，屬於頭域的組成部分，User Agent也簡稱UA

weixin_42929804 2020-07-07 03:33:36

五、分佈式爬蟲學習之BeautSoup4

BeautifulSoup4庫 和lxml一樣，BeautifulSoup也是一個HTML/XML的解析器，主要功能也是如何解析和提取HTML/XML數據。 區別：lxml只會局部遍歷，而BeautifulSoup是基於HTML

Mr_Little_li 2020-07-06 17:47:01

十三、學習分佈式爬蟲之字體反爬

字體反爬 字體反爬原理 網頁開發者自己創造一種字體，因爲在字體中每個文字都有其代號，那麼以後在網頁中不會直接顯示這個文字的最終的效果，而是顯示他的代號，因此即使獲取到了網頁中的文本內容，也只是獲取到文字的代號，而不是文字本身。

Mr_Little_li 2020-07-06 17:47:01

八、學習分佈式爬蟲之多線程

多線程爬蟲 理解多線程 掌握threading模塊的使用 掌握生產者消費者模式 理解GIL 能用多線程寫爬蟲 什麼是多線程 理解：默認情況下，一個程序只有一個進程和一個線程，代碼是依次線性執行的，而多線程則可以併發執行，一次性多

Mr_Little_li 2020-07-06 17:47:01

十四、學習分佈式爬蟲之Scrapy

Scrapy框架 學習目標 理解scrapy框架。 學會spider爬蟲的編寫。 學會Crawlspider爬蟲編寫。 學會中間件的編寫。 學會pipeline保存數據。 學會將Scrapy結合selenium一起使用。 學會在

Mr_Little_li 2020-07-06 17:47:01