獲取拉勾網招聘信息數據

使用 curl 模擬 ajax 請求獲取數據

1、找到目標城市和目標職位，然後按“最新排序”。

鏈接爲：https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC#order

2、右鍵點擊頁面，選擇"檢查"，進入瀏覽器調試界面，切換到調試器的 network -> xhr 標籤下

3、cmd + R 刷新頁面，此時會捕捉到此頁面發出的xhr請求，找到 positionAjax.json 開頭的請求，右鍵點擊選擇 Copy as cUrl(bash)。

4、本文是wins，安裝curl

下載curl，本文使用的是：curl-7.33.0-win64-ssl-sspi.zip

下載地址爲：http://download.csdn.net/detail/u012689336/9683328

下載好後解壓到桌面就好了。

5、安裝git

本文使用的是：Git-2.9.3-64-bit.exe

下載地址爲：http://download.csdn.net/detail/u012689336/9683333

下載之後直接安裝就可以了。

6、桌面右鍵打開，選擇Git Bash Here

7、上一步中的curl指令複製到終端，即右鍵選擇粘貼。

curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; HISTORY_POSITION=2330069%2C15k-25k%2CUdesk%EF%BC%8D%E4%BC%81%E4%B8%9A%E7%BA%A7%E6%99%BA%E8%83%BD%E5%AE%A2%E6%9C%8D%E5%B9%B3%E5%8F%B0%2CJava%7C; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; _gat=1; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGSID=20161115133620-703f2d88-aaf5-11e6-a7f9-525400f775ce; LGRID=20161115133840-c33e5bf7-aaf5-11e6-a7fa-525400f775ce; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447,1479179434,1479188155,1479188294; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479188294; _ga=GA1.2.1088550872.1479178314; SEARCH_ID=12f8eb7a0d384e5eba66ed04f73c4cdf' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' --data 'first=true&pn=1&kd=iOS' --compressed

這個 curl代碼非常長，對於本次分析來說，最關鍵的是末尾的 pn=1&kd=iOS，分別代表着頁面和職位，動態設置，即可獲取更多職位的更多數據。

8、按下回車鍵，即可看到輸出。可以看到，與網站的第一頁獲取的實際數據是完全對應的。

{"success":true,"requestId":null,"resubmitToken":null,"msg":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"resultSize":15,"locationInfo":{"city":"北京","district":null,"queryByGisCode":false,"businessZone":null,"locationCode":null},"queryAnalysisInfo":{"positionName":"ios","companyName":null,"industryName":null,"usefulCompany":false},"strategyProperty":{"name":"dm-csearch-newSimScorer","id":1},"totalCount":805,"result":[{"businessZones":null,"imState":"today","lastLogin":1479189411000,"publisherId":3003531,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"開發/測試/運維類","secondType":"前端開發/移動開發","positionLables":null,"createTime":"2016-11-15 13:58:23","companyId":101,"positionName":"WX-iOS開發工程師","education":"本科","city":"北京","positionId":2578379,"financeStage":"上市公司","companyShortName":"迅雷","companyLogo":"image1/M00/00/01/Cgo8PFTUV_SADj9rAABrXBj73Q8580.jpg","salary":"15k-30k","industryField":"移動互聯網","district":null,"companyLabelList":["年底雙薪","節日禮物","技能培訓","免費班車"],"positionAdvantage":"五險一金+班車+早晚餐+健康體檢+運動日","jobNature":"全職","workYear":"3-5年","approve":1,"score":0,"companySize":"500-2000人","companyFullName":"深圳市迅雷網絡技術有限公司","adWord":0,"formatCreateTime":"13:58發佈"},{"businessZones":["立水橋","北苑"],"imState":"today","lastLogin":1479188246000,"publisherId":5071165,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"前端開發/移動開發","positionLables":["平臺","iOS","遊戲"],"createTime":"2016-11-15 13:54:42","companyId":129516,"positionName":"iOS遊戲平臺接入SDK開發工程師","education":"不限","city":"北京","positionId":1906251,"financeStage":"成長型(A輪)","companyShortName":"天遊互動","companyLogo":"i/image/M00/2B/8B/CgqKkVc1RVWATTcRAABOmw2czME554.jpg","salary":"5k-10k","industryField":"移動互聯網,遊戲","district":"朝陽區","companyLabelList":["移動遊戲","專項獎金","午餐補助","通訊津貼"],"positionAdvantage":"彈性工作，可享期權，項目獎金提成","jobNature":"全職","workYear":"1-3年","approve":0,"score":0,"companySize":"50-150人","companyFullName":"北京天遊互動科技有限公司","adWord":0,"formatCreateTime":"13:54發佈"},{"businessZones":["知春路","中關村","雙榆樹"],"imState":"today","lastLogin":1479188052000,"publisherId":208,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"前端開發/移動開發","positionLables":["iOS"],"createTime":"2016-11-15 13:34:44","companyId":139,"positionName":"iOS","education":"本科","city":"北京","positionId":1941892,"financeStage":"成長型(A輪)","companyShortName":"九玉","companyLogo":"image1/M00/00/02/Cgo8PFTUV_WAD7hfAABXYsAPI8k943.png","salary":"15k-25k","industryField":"移動互聯網","district":"海淀區","companyLabelList":["技能培訓","股票期權","帶薪年假","崗位晉升"],"positionAdvantage":"股票期權,彈性工作制,免費工作餐,水果零食","jobNature":"全職","workYear":"不限","approve":1,"score":0,"companySize":"15-50人","companyFullName":"九玉（北京）科技有限公司","adWord":0,"formatCreateTime":"13:34發佈"},{"businessZones":["望京","大山子","酒仙橋"],"imState":"today","lastLogin":1479187427000,"publisherId":1493277,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"移動開發","positionLables":["iOS"],"createTime":"2016-11-15 13:25:43","companyId":59119,"positionName":"iOS高級開發工程師","education":"大專","city":"北京","positionId":1216114,"financeStage":"成長型(A輪)","companyShortName":"君融貸","companyLogo":"i/image/M00/21/36/Cgp3O1cQU_-ADtiUAAAaLx-Z_J0830.jpg","salary":"15k-25k","industryField":"金融","district":"朝陽區","companyLabelList":["節日禮物","帶薪年假","崗位晉升","管理規範"],"positionAdvantage":"帶薪年假 彈性工作 領導好","jobNature":"全職","workYear":"5-10年","approve":1,"score":0,"companySize":"50-150人","companyFullName":"君融貸（北京）信息技術服務有限公司","adWord":0,"formatCreateTime":"13:25發佈"},{"businessZones":["新街口","德勝門","小西天"],"imState":"today","lastLogin":1479189406000,"publisherId":2647983,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"開發/測試/運維類","secondType":"前端開發/移動開發","positionLables":["iOS"],"createTime":"2016-11-15 13:18:13","companyId":97172,"positionName":"iOS開發工程師","education":"本科","city":"北京","positionId":2578159,"financeStage":"初創型(天使輪)","companyShortName":"Humax Tech","companyLogo":"image2/M00/07/B8/CgpzWlYAG72AQz4dAAAWnjtTGgg550.jpg?cc=0.3403482724724128","salary":"10k-18k","industryField":"硬件","district":"西城區","companyLabelList":["年終分紅","交通補助","午餐補助","績效獎金"],"positionAdvantage":"前景好,晉升空間,彈性","jobNature":"全職","workYear":"3-5年","approve":1,"score":0,"companySize":"15-50人","companyFullName":"合優智景（北京）科技有限公司","adWord":0,"formatCreateTime":"13:18發佈"},{"businessZones":null,"imState":"disabled","lastLogin":1479185306000,"publisherId":6082811,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"移動開發","positionLables":["iOS","架構師"],"createTime":"2016-11-15 12:53:55","companyId":10723,"positionName":"iOS架構師","education":"本科","city":"北京","positionId":2535611,"financeStage":"成長型(B輪)","companyShortName":"酷劃在線","companyLogo":"image1/M00/00/15/CgYXBlTUWEOAUTXzAABQJHYVxSk792.jpg","salary":"20k-35k","industryField":"移動互聯網","district":null,"companyLabelList":["國際公司","崗位晉升","扁平管理","美女多"],"positionAdvantage":"平臺大 牛人多 技術氛圍好 14薪","jobNature":"全職","workYear":"5-10年","approve":1,"score":0,"companySize":"50-150人","companyFullName":"北京酷劃在線網絡技術有限公司","adWord":0,"formatCreateTime":"12:53發佈"},{"businessZones":["五道口","學院路","北航"],"imState":"today","lastLogin":1479181759000,"publisherId":5561461,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"移動開發","positionLables":["iOS"],"createTime":"2016-11-15 12:11:27","companyId":51665,"positionName":"iOS","education":"本科","city":"北京","positionId":2501368,"financeStage":"成長型(A輪)","companyShortName":"樂動力","companyLogo":"image1/M00/3A/9F/Cgo8PFWt-HGAZhiDAAAb-JIwkf4960.jpg","salary":"15k-25k","industryField":"移動互聯網","district":"海淀區","companyLabelList":["工程師文化","股票期權","年度旅遊","扁平管理"],"positionAdvantage":"千萬級用戶量，彈性工作制，13薪期權獎金","jobNature":"全職","workYear":"1-3年","approve":1,"score":0,"companySize":"15-50人","companyFullName":"北京朗動科技有限公司","adWord":0,"formatCreateTime":"12:11發佈"},{"businessZones":["上地","西二旗","馬連窪"],"imState":"today","lastLogin":1479180359000,"publisherId":3616515,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"前端開發/移動開發","positionLables":["iOS"],"createTime":"2016-11-15 11:59:45","companyId":7292,"positionName":"iOS","education":"本科","city":"北京","positionId":1635518,"financeStage":"成長型(B輪)","companyShortName":"暢聊天下(派派交友）","companyLogo":"image1/M00/1A/42/CgYXBlUbifGASvN9AACTHYLzJKc557.jpg","salary":"15k-30k","industryField":"移動互聯網","district":"海淀區","companyLabelList":["年底雙薪","午餐補助","帶薪年假","股票期權"],"positionAdvantage":"期權期權期權！！！","jobNature":"全職","workYear":"3-5年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京暢聊天下科技有限公司","adWord":0,"formatCreateTime":"11:59發佈"},{"businessZones":["五道口","學院路","北航"],"imState":"today","lastLogin":1479181767000,"publisherId":1544460,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"前端開發/移動開發","positionLables":["iOS"],"createTime":"2016-11-15 11:49:32","companyId":37236,"positionName":"高級IOS開發工程師","education":"本科","city":"北京","positionId":2195782,"financeStage":"成熟型(C輪)","companyShortName":"鬥魚直播","companyLogo":"image1/M00/1E/66/Cgo8PFUuJ7SAcErwAACtNjsgHHk099.jpg","salary":"15k-25k","industryField":"移動互聯網","district":"海淀區","companyLabelList":["技能培訓","績效獎金","年度旅遊","管理規範"],"positionAdvantage":"優厚的薪酬和績效激勵，全年16-18薪","jobNature":"全職","workYear":"3-5年","approve":1,"score":0,"companySize":"500-2000人","companyFullName":"武漢鬥魚網絡科技有限公司","adWord":0,"formatCreateTime":"11:49發佈"},{"businessZones":["小營","對外經貿","芍藥居"],"imState":"today","lastLogin":1479187254000,"publisherId":633272,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"開發/測試/運維類","secondType":"前端開發/移動開發","positionLables":null,"createTime":"2016-11-15 11:46:01","companyId":6731,"positionName":"iOS高級開發","education":"本科","city":"北京","positionId":2577834,"financeStage":"成長型(不需要融資)","companyShortName":"有緣網","companyLogo":"i/image/M00/54/7F/CgqKkVfD9yqAJ5QyAAAypAzMizU130.jpg","salary":"15k-20k","industryField":"移動互聯網,社交網絡","district":"朝陽區","companyLabelList":["績效獎金","年終分紅","股票期權","年底雙薪"],"positionAdvantage":"更專注的開發工作","jobNature":"全職","workYear":"3-5年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京友緣在線網絡科技有限責任公司","adWord":0,"formatCreateTime":"11:46發佈"},{"businessZones":["蘇州街","萬泉河","頤和園"],"imState":"today","lastLogin":1479183814000,"publisherId":7931,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"前端開發/移動開發","positionLables":["iOS"],"createTime":"2016-11-15 11:40:07","companyId":1249,"positionName":"IOS Lead","education":"本科","city":"北京","positionId":2347426,"financeStage":"成熟型(C輪)","companyShortName":"出門問問","companyLogo":"image1/M00/40/3A/CgYXBlXC_4OASRgsAAAislBu0uU808.jpg","salary":"30k-40k","industryField":"移動互聯網,O2O","district":"海淀區","companyLabelList":["帥哥多","美女多","績效獎金","股票期權"],"positionAdvantage":"谷歌技術氛圍，行業技術大咖帶隊","jobNature":"全職","workYear":"5-10年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京羽扇智信息科技有限公司","adWord":0,"formatCreateTime":"11:40發佈"},{"businessZones":["大山子","酒仙橋","將臺路"],"imState":"today","lastLogin":1479179230000,"publisherId":5569772,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"移動開發","positionLables":["iOS"],"createTime":"2016-11-15 11:31:27","companyId":114110,"positionName":"iOS開發實習生","education":"碩士","city":"北京","positionId":2483585,"financeStage":"上市公司","companyShortName":"樂視","companyLogo":"i/image/M00/04/2B/Cgp3O1bGwzWAEAB4AAAbnMQCudk783.jpg","salary":"4k-5100 15675    0 15653  100    22  34630     48 --:--:-- --:--:-- --:--:-- 35819期權","交通補助","通訊津貼","帶薪年假"],"positionAdvantage":"轉正機會,有競爭力的待遇,期權,節日福利","jobNature":"實習","workYear":"應屆畢業生","approve":1,"score":0,"companySize":"2000人以上","companyFullName":"樂視網信息技術（北京）股份有限公司","adWord":0,"formatCreateTime":"11:31發佈"},{"businessZones":null,"imState":"today","lastLogin":1479189156000,"publisherId":127369,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"移動開發","positionLables":["iOS"],"createTime":"2016-11-15 11:13:16","companyId":10007,"positionName":"高級iOS開發工程師","education":"本科","city":"北京","positionId":1831196,"financeStage":"成長型(A輪)","companyShortName":"野獸騎行","companyLogo":"image1/M00/11/C9/CgYXBlUBXjuARqvEAAB5Sv6Sr1Q235.jpg","salary":"20k-28k","industryField":"移動互聯網,硬件","district":null,"companyLabelList":["年底雙薪","午餐補助","定期體檢","股票期權"],"positionAdvantage":"股票期權、年底雙薪、年終獎、帶薪休假","jobNature":"全職","workYear":"3-5年","approve":1,"score":0,"companySize":"50-150人","companyFullName":"北京野獸科技有限公司","adWord":0,"formatCreateTime":"11:13發佈"},{"businessZones":null,"imState":"today","lastLogin":1479179248000,"publisherId":7123,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"移動開發","positionLables":["iOS"],"createTime":"2016-11-15 11:07:44","companyId":546,"positionName":"iOS Leader","education":"本科","city":"北京","positionId":1927349,"financeStage":"成熟型(C輪)","companyShortName":"V電影","companyLogo":"image1/M00/00/03/Cgo8PFTUV_uAcfZsAAAxyPH5hXQ445.jpg","salary":"30k-50k","industryField":"移動互聯網,文化娛樂","district":null,"companyLabelList":["專項獎金","節日禮物","崗位晉升","領導好"],"positionAdvantage":"提供早餐，加班晚餐，零食飲料、期權","jobNature":"全職","workYear":"5-10年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京新片場傳媒股份有限公司","adWord":0,"formatCreateTime":"11:07發佈"},{"businessZones":["望京","大山子","花家地"],"imState":"today","lastLogin":1479188721000,"publisherId":1467174,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技術","secondType":"前端開發/移動開發","positionLables":["iOS","架構師"],"createTime":"2016-11-15 11:05:23","companyId":59251,"positionName":"ios架構師","education":"本科","city":"北京","positionId":2476028,"financeStage":"成長型(B輪)","companyShortName":"映客直播","companyLogo":"i/image/M00/08/49/Cgp3O1bPycGAMj0hAAAgshBCvv4840.jpg","salary":"40k-55k","industryField":"移動互聯網,文化娛樂","district":"朝陽區","companyLabelList":["年底雙薪","專項獎金","股票期權","帶薪年假"],"positionAdvantage":"福利完善,年底獎金,環境優美,帥哥美女","jobNature":"全職","workYear":"3-5年","approve":1,"score":0,"companySize":"500-2000人","companyFullName":"北京蜜萊塢網絡科技有限公司","adWord":0,"formatCreateTime":"11:05發佈"}]}},"code":0}

如何將數據保存爲文件?
將curl的結果，直接保存爲文件，才方便進一步處理，方法就是使用重定向符 >，是保存到指定文件 1.json

$ curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178314,1479178447; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447; _ga=GA1.2.1088550872.1479178314; LGSID=20161115105219-86a4c8d9-aade-11e6-98ea-5254005c3644; LGRID=20161115105432-d5bbfdde-aade-11e6-a55e-525400f775ce; SEARCH_ID=baf7de67edbc4524b3e39f79042601e9' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data 'first=true&pn=1&kd=iOS' --compressed > 1.json

如何獲取其他職位的數據?
需要一個for in 循環來遍歷一組給定的職位，動態更改前面curl腳本中的末尾的kd屬性的值，並寫入職位對應的文件中，注意：末尾 --data後的單引號對，要改成雙引導對，否則無法應用變量。完整代碼如下,職位數組,可按需自行添加:

$ for ssd in "Java" "PHP" "C" "C++" "Android" "iOS"

do 

curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178314,1479178447; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447; _ga=GA1.2.1088550872.1479178314; LGSID=20161115105219-86a4c8d9-aade-11e6-98ea-5254005c3644; LGRID=20161115105432-d5bbfdde-aade-11e6-a55e-525400f775ce; SEARCH_ID=baf7de67edbc4524b3e39f79042601e9' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=1&kd=$ssd" --compressed > $ssd.json

done

如何批量獲取?
curl 腳本，現在是每次只可以獲取單頁，要想獲取多頁，加個for循環就可以了。經過觀察，拉勾有效數據大概最多在100頁左右，所以寫個1~100的循環，並以 $kd_$pn.json 的格式保存:

for (( pn=1; pn<=100; pn=pn+1 )); do

for ssd in "Java" "PHP" "C" "C++" "Android" "iOS"

do 

curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178314,1479178447; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447; _ga=GA1.2.1088550872.1479178314; LGSID=20161115105219-86a4c8d9-aade-11e6-98ea-5254005c3644; LGRID=20161115105432-d5bbfdde-aade-11e6-a55e-525400f775ce; SEARCH_ID=baf7de67edbc4524b3e39f79042601e9' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=1&kd=$ssd" --compressed > $ssd\_$pn.json

done

done