假期在家撸了一个 python 爬虫

Traceback (most recent call last):
  File "alexa.py", line 116, in <module>
    site.multiprocess()
  File "alexa.py", line 83, in multiprocess
    self.domain = multiprocessing.Manager().list()
  File "/home/rocky/python/lib/python2.7/multiprocessing/managers.py", line 667, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/home/rocky/python/lib/python2.7/multiprocessing/managers.py", line 565, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/home/rocky/python/lib/python2.7/multiprocessing/connection.py", line 175, in Client
    answer_challenge(c, authkey)
  File "/home/rocky/python/lib/python2.7/multiprocessing/connection.py", line 432, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
IOError: [Errno 11] Resource temporarily unavailable

Python

file

line

rocky

15 条回复 • 2016-10-09 09:34:51 +08:00

zwpaper

2016-10-07 14:23:18 +08:00 via iPhone

手机没细看，瞎猜一下，是不是 Socket 端口占用没处理好

hellorocky728

2016-10-07 14:35:32 +08:00

@zwpaper 应该不是吧,我都没有起服务,没有占用端口

ericls

2016-10-07 14:57:10 +08:00

试试 loop.run_in_executor 呢？用 future 的形式来看看

firebroo

2016-10-07 15:04:20 +08:00 via Android

resource xxx 报错应该是进程太多了

stranbird

2016-10-07 15:49:22 +08:00

可以先用 pep8 lint 一下。

pright

2016-10-07 15:51:22 +08:00

http://bugs.python.org/issue6056

multiprocessing 的坑感觉有点多

prasanta

2016-10-07 22:46:27 +08:00

为什么不用 aasync/await 呢

panda0

2016-10-08 10:05:28 +08:00

https://www.zhihu.com/question/29927678?sort=created

hellorocky728

2016-10-08 10:17:29 +08:00

@prasanta 这两天打算学学,然后用上,比较一下~

rale

2016-10-08 10:42:28 +08:00

这个 errno 11 来自于 c 语言， http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/Errors/unix_system_errors.html ，按这里的 define 你重试几次看看。

hellorocky728

2016-10-08 17:21:19 +08:00

@rale 虽然不知道怎么做,但还是感谢~

hellorocky728

2016-10-08 17:21:37 +08:00

@stranbird 谢谢

ToughGuy

2016-10-08 17:29:29 +08:00

ulimit -a 看看

hellorocky728

2016-10-08 18:05:16 +08:00

@ToughGuy

`
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3862
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 10240
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3862
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

`

ToughGuy

2016-10-09 09:34:51 +08:00

系统资源限制问题不大，你代码里面多进程部分最好加上 queue, 限制同时 fork 的进程数量。