用Request爬取實戰

request.Request類

如果想要在請求的時候增加一些請求頭，用request.Request。比如要增加一個User-Agent

在拉勾網的職業信息通過另外的網址，再通過JS嵌入到主頁面的html代碼當中的
真正的職業信息

在json.cn中解碼得

請求頁面，還有請求方式爲POST

from urllib import request
url="https://www.lagou.com/jobs/positionAjax.json?px=default&gx=%E5%AE%9E%E4%B9%A0&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false&isSchoolJob=1"
#更改頭部信息
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36','Referer':"https://www.lagou.com/jobs/list_python%E7%88%AC%E8%99%AB/p-city_0?px=default&gx=%E5%85%A8%E8%81%8C&gj=&xl=%E6%9C%AC%E7%A7%91&isSchoolJob=1"}
data={'first':'true','pn':1,'kd':'python'}
req = request.Request(url,headers = headers,data=data,method='POST')
resp = request.urlopen(req)
print(resp.read())

錯誤顯示data得數據類型不對則還要對其進行編碼，引進parse得urlencode方法

from urllib import request,parse
url="https://www.lagou.com/jobs/positionAjax.json?px=default&gx=%E5%AE%9E%E4%B9%A0&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false&isSchoolJob=1"
#更改頭部信息
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36','Referer':"https://www.lagou.com/jobs/list_python%E7%88%AC%E8%99%AB/p-city_0?px=default&gx=%E5%85%A8%E8%81%8C&gj=&xl=%E6%9C%AC%E7%A7%91&isSchoolJob=1"}
data={'first':'true','pn':1,'kd':'python'}
req = request.Request(url,headers = headers,data=parse.urlencode(data),method='POST')
resp = request.urlopen(req)
print(resp.read())

POST數據應該爲編碼後的數據，python3中默認數據類型爲utf-8類型，通過encode函數

from urllib import request,parse
url="https://www.lagou.com/jobs/positionAjax.json?px=default&gx=%E5%AE%9E%E4%B9%A0&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false&isSchoolJob=1"
#更改頭部信息
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36','Referer':"https://www.lagou.com/jobs/list_python%E7%88%AC%E8%99%AB/p-city_0?px=default&gx=%E5%85%A8%E8%81%8C&gj=&xl=%E6%9C%AC%E7%A7%91&isSchoolJob=1"}
data={'first':'true','pn':1,'kd':'python'}
req = request.Request(url,headers = headers,data=parse.urlencode(data).encode('utf-8'),method='POST')
resp = request.urlopen(req)
print(resp.read())

返回一個錯誤信息，前面有一個b表示爲byts數據類型解碼decode()成utf-8類型

#解碼decode（）
print(resp.read().decode('utf-8'))

因爲用瀏覽器請求頁面可以正常得到數據，則顯而易見是電腦破解你是爬蟲程序
頁面查找ctrl+F
顯示函數代碼 ctrl+B