爬蟲-鏈接驗證

爬蟲-鏈接驗證

@(Python)[python]

鏈接驗證

編寫一個程序，對給定的網頁URL，下載該頁面所有鏈接的頁面。程序應該標記出所有具有404“Not Found”狀態碼的頁面，將它們作爲壞鏈接輸出。

import requests
import bs4

url = 'http://ifeve.com/'
response = requests.get(url)
soup = bs4.BeautifulSoup(response.text, "lxml")
a_list = soup.select('a')
for a in a_list:
    a_url = a.get('href')
    try:
        response = requests.get(a_url)
        if response.status_code == requests.codes.not_found:
            print("Page %s is broken link" % a_url)
        else:
            print("Page %s is other type link" % a_url)
        response.raise_for_status()
    except:
        print("Page %s is Error" % a_url)