CentOS 下 Python web server 出现 too many open files 异常如何排查问题？

[zeroday]

在 linux 下用 python+ansible+django 写的网站，每隔一段时间就会出现 too many open files 的异常。

我该如何排查这个问题呢？

[shyling]

limits 设高点

[wzxjohn]

ulimit -a

[zeroday]

@wzxjohn

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 126875
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 200000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 126875
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[zeroday]

@shyling 已经设置成 200000

[zeroday]

@shyling 已经设置为 200000 了，还是会满。

[BOYPT]

得从你的程序分析，是什么打开了描述字，什么导致没有回收。

[ryd994]

用 lsof -p python 的 pid 看看都打开了什么文件

[zeroday]

@BOYPT @ryd994

查了一下文件已经删除，但是没有释放的文件描述符。

[root@85-13-112]# lsof | grep deleted
python 19327 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19332 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19332 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19334 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19334 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19335 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19335 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19336 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19336 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19337 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19337 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19338 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19338 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19339 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19339 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19340 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19340 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19341 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19341 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19342 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19342 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19343 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19343 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19344 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19344 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19345 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19345 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19346 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19346 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19347 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19347 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19348 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19348 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19349 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19349 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19350 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19350 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19351 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19351 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19352 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19352 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19353 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19353 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19354 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19354 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19355 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19355 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19356 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19356 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19357 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19357 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19358 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19358 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19359 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19359 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19360 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19360 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19361 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19361 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19362 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19362 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)
python 19327 19363 root 8u REG 252,2 0 534767 /tmp/tmp4Hlr9O (deleted)
python 19327 19363 root 12u REG 252,2 0 534768 /tmp/tmp2TSn03 (deleted)

[root@85-13-112]# lsof | grep deleted | wc -l
65

[ryd994]

@zeroday 所以用完文件随手关一下啊

[BOYPT]

@zeroday 比如说使用了 tempfile.TemporaryFile 返回的描述字对象一直没有 close ，就这样了啊

[zeroday]

@BOYPT 谢谢，后来我也发现了 ansible 里这段代码。

```
def write_file(module, url, dest, content):
# create a tempfile with some test content
fd, tmpsrc = tempfile.mkstemp()
f = open(tmpsrc, 'wb')
try:
f.write(content)
except Exception, err:
os.remove(tmpsrc)
module.fail_json(msg="failed to create temporary content file: %s" % str(err))
f.close()
```

fd, tmpsrc = tempfile.mkstemp() 生成的 fd 没有 close 。

还有一个问题，就是发现线上服务器上 /dev/null 这个描述符。这个可能是什么没有关呢？

[BOYPT]

@zeroday 一般服务进程的的 stdin/stdout/stderr 三个特殊描述字，可能会指向 /dev/null ，不需要关闭。