关于爬取知乎大 V 关注者的信息,遇到了点障碍,求大神前来指教!

[yamiedie]

如图,比如说我要爬大头死变态的关注人数 已知 URL 为:https://www.zhihu.com/people/wang-ni-ma-94/followers?page=1 通过抓包发现网站信息是动态加载的,然后顺藤摸瓜找到了对应的疑似 json 文件 api 接口为: https://www.zhihu.com/api/v4/members/wang-ni-ma-94/followers?include=data%5B*%5D.answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F(type%3Dbest_answerer)%5D.topics&offset=20&limit=20

在浏览器打开后是形如下的一段文本,疑似 json:

// 20180122152224 // https://www.zhihu.com/api/v4/members/wang-ni-ma-94/followers?include=data%5B*%5D.answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F(type%3Dbest_answerer)%5D.topics&offset=20&limit=20 { "paging": { "is_end": false, "totals": 614311, "previous": "http://www.zhihu.com/api/v4/members/wang-ni-ma-94/followers?include=data%5B%2A%5D.answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F%28type%3Dbest_answerer%29%5D.topics&limit=20&offset=0", "is_start": false, "next": "http://www.zhihu.com/api/v4/members/wang-ni-ma-94/followers?include=data%5B%2A%5D.answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F%28type%3Dbest_answerer%29%5D.topics&limit=20&offset=40" }, "data": [ { "is_followed": false, "avatar_url_template": "https://pic2.zhimg.com/9256b7b963c816a5b5cae6b4b9b67bcf_{size}.jpg", "user_type": "people", "answer_count": 0, "is_following": false, "url": "http://www.zhihu.com/api/v4/people/44ced3e1aa77f2aac54447cc5219e523", "type": "people", "url_token": "zack-71-48", "id": "44ced3e1aa77f2aac54447cc5219e523", "articles_count": 0, "name": "zack", "headline": "", "gender": -1, "is_advertiser": false, "avatar_url": "https://pic2.zhimg.com/9256b7b963c816a5b5cae6b4b9b67bcf_is.jpg", "is_org": false, "follower_count": 0, "badge": [ ...... 之所以说疑似,是因为我在使用 requests.get(url).json()之后报错: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

网上搜一了一下,说有可能不是合法的 json 文件 有哪位大兄 dei 知道怎么破吗? ps:因为这网站不能发图片,所以排版会丑一点,大家别见怪哈!

[yamiedie]

排版像坨 shi 啊,可是已经编辑不了了,大神们将就看着吧

[1130335361]

直接 requests.get(url).json()是不行的，你打印 requests.get(url).content 看看返回的是什么

需要加上 User-Agent 和 cookies 去请求

[1130335361]

不用 cookie 好像也行
https://gist.github.com/anonymous/dee2672188fb65cd68f2f00132fa5f4b

[yamiedie]

@1130335361 请求前的准备我做完了,是没问题的,我检查了下,是因为刚开始的那个 // 20180122152224 的锅,因为这个的存在,这就不叫 json 文件了,只能是有一定规律的文件.现在问题是该怎么处理那个文件,好让它能让我反序列化.

[yamiedie]

@1130335361 连接挂了,好像....

[1130335361]

@yamiedie 链接好的 要翻过去看

[yamiedie]

@1130335361 晚上才有梯子,总之,谢谢你了!

[yamiedie]

@1130335361 打印 response.content 是一串很长的字节流 b'....',但是我 decode 依然出错:invalid continuation byte...

[yamiedie]

@1130335361 问题解决了,看了你的 GitHub,豁然开朗!非常感谢!