使用 pycharm 对 Python 项目调试遇到非常诡异的现象

2019-03-13 21:47:12 +08:00
 hanssx

调试项目： https://github.com/WhaleShark-Team/cobra

脚本命令：python cobra.py -t /root/php/dvwa/vulnerabilities/sqli_blind/ -d

我在公司开发机上面 pipenv 搭建使用的是 py3.7，一开始我是用的 pycharm 远程调试，Run 没有任何问题，但是 Debug 会在进程池代码 multiprocessing.Pool()附近报错 KeyboardInterrupt 吗，实际上我什么也没有做，Run 没事儿，Debug 点一下就会自动中断。

ssh://root@10.26.15.222:22/root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python -u /root/.pycharm_helpers/pydev/pydevd.py --multiproc --qt-support --client '0.0.0.0' --port 40819 --file /root/python/cobra/cobra.py -t /root/php/dvwa/vulnerabilities/sqli_blind/ -d
warning: Debugger speedups using cython not found. Run '"/root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python" "/root/.pycharm_helpers/pydev/setup_cython.py" build_ext --inplace' to build.
pydev debugger: process 26949 is connecting

Connected to pydev debugger (build 171.4694.94)
[21:37:52] [DEBUG] [INIT] set logging level: debug
[21:37:52] [DEBUG] [INIT] start scanning...
[21:37:52] [INFO] [REPORT] Report URL: ?sid=a7df1eil4boi
[21:37:52] [DEBUG] [PARSE-ARGS] Target Mode: folder
[21:37:52] [DEBUG] [PARSE-ARGS] Output Mode: stream
[21:37:52] [DEBUG] [PARSE-ARGS] target directory: /root/php/dvwa/vulnerabilities/sqli_blind/
[21:37:52] [INFO] [CLI] Target directory: /root/php/dvwa/vulnerabilities/sqli_blind
[21:37:52] [DEBUG] [PICKUP] /root/php/dvwa/vulnerabilities/sqli_blind
[21:37:52] [DEBUG] [PICKUP] [FILES] |--index.php
[21:37:52] [DEBUG] [PICKUP] [FILES] |--source
[21:37:52] [DEBUG] [PICKUP] [FILES] |  |--medium.php
[21:37:52] [DEBUG] [PICKUP] [FILES] |  |--impossible.php
[21:37:52] [DEBUG] [PICKUP] [FILES] |  |--low.php
[21:37:52] [DEBUG] [PICKUP] [FILES] |  |--high.php
[21:37:52] [DEBUG] [PICKUP] [FILES] |--help
[21:37:52] [DEBUG] [PICKUP] [FILES] |  |--help.php
[21:37:52] [DEBUG] [PICKUP] [FILES] |--cookie-input.php
[21:37:52] [DEBUG] [PICKUP] [EXTENSION-COUNT] .php : 7
[21:37:52] [DEBUG] [DETECTION] [LANGUAGE] .php 7
[21:37:52] [DEBUG] [DETECTION] [LANGUAGE] found the chiefly language(php), maybe have largest, continue...
[21:37:52] [DEBUG] [DETECTION] [LANGUAGE] main language(php), tmp language(None)
[21:37:52] [DEBUG] Dependency analysis cannot be done without finding dependency files
[21:37:52] [INFO] [DETECTION] [FRAMEWORK] Unknown Framework
[21:37:52] [INFO] [CLI] [STATISTIC] Language: php Framework: Unknown Framework
[21:37:52] [INFO] [CLI] [STATISTIC] Files: 7, Extensions:1, Consume: 0.0
[21:37:52] [DEBUG] filter dependency rules
[21:37:52] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:53] [DEBUG] filter dependency rules
[21:37:54] [DEBUG] filter dependency rules
[21:37:54] [DEBUG] filter dependency rules
[21:37:54] [DEBUG] filter dependency rules
[21:37:54] [DEBUG] filter dependency rules
[21:37:54] [DEBUG] filter dependency rules
[21:37:54] [DEBUG] filter dependency rules
[21:37:54] [INFO] [PUSH] 18 CVE Rules
[21:37:55] [DEBUG] Dependency analysis cannot be done without finding dependency files
[21:37:55] [DEBUG] [SCAN] [STORE] Not found vulnerabilities on this rule!
^CTraceback (most recent call last):
  File "/root/.pycharm_helpers/pydev/pydevd.py", line 1591, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/root/.pycharm_helpers/pydev/pydevd.py", line 1018, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/root/python/cobra/cobra.py", line 22, in <module>
    sys.exit(main())
  File "/root/python/cobra/cobra/__init__.py", line 110, in main
    cli.start(args.target, args.format, args.output, args.special_rules, a_sid, args.dels)
  File "/root/python/cobra/cobra/cli.py", line 94, in start
    language=main_language, framework=main_framework, file_count=file_count, extension_count=len(files))
  File "/root/python/cobra/cobra/engine.py", line 160, in scan
    cve_vuls = scan_cve(target_directory)
  File "/root/python/cobra/cobra/cve.py", line 406, in scan_cve
    pool.join()
  File "/usr/local/python3/lib/python3.7/multiprocessing/pool.py", line 560, in join
    p.join()
  File "/usr/local/python3/lib/python3.7/multiprocessing/process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "/usr/local/python3/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/usr/local/python3/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

Process finished with exit code 1

因为看不懂报错代码，所以我就使用了我搭建在 vmware 中的虚拟机 centos7，然后同样使用 pipenv 安装了 py3.7 的虚拟环境，但是诡异得是 Run 都会产生问题了，而且报错也和之前远程调试 Debug 不一样，

/root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python /root/python/cobra/cobra.py -t /root/python/cobra/tests/vulnerabilities/
--- Logging error ---
Traceback (most recent call last):
  File "/root/python/cobra/cobra/log.py", line 123, in emit
    stream.write(message)
TypeError: write() argument must be str, not bytes
Call stack:
  File "/root/python/cobra/cobra.py", line 22, in <module>
    sys.exit(main())
  File "/root/python/cobra/cobra/__init__.py", line 110, in main
    cli.start(args.target, args.format, args.output, args.special_rules, a_sid, args.dels)
  File "/root/python/cobra/cobra/cli.py", line 64, in start
    logger.info('[REPORT] Report URL: {u}'.format(u=report))
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1332, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1468, in _log
    self.handle(record)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1478, in handle
    self.callHandlers(record)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1540, in callHandlers
    hdlr.handle(record)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 854, in handle
    self.emit(record)
  File "/root/python/cobra/cobra/log.py", line 134, in emit
    self.handleError(record)
Message: '[REPORT] Report URL: ?sid=a9d766qddopy'
Arguments: ()
--- Logging error ---
Traceback (most recent call last):
  File "/root/python/cobra/cobra/log.py", line 123, in emit
    stream.write(message)
TypeError: write() argument must be str, not bytes
Call stack:
  File "/root/python/cobra/cobra.py", line 22, in <module>
    sys.exit(main())
  File "/root/python/cobra/cobra/__init__.py", line 110, in main
    cli.start(args.target, args.format, args.output, args.special_rules, a_sid, args.dels)
  File "/root/python/cobra/cobra/cli.py", line 75, in start
    logger.info('[CLI] Target directory: {d}'.format(d=target_directory))
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1332, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1468, in _log
    self.handle(record)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1478, in handle
    self.callHandlers(record)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1540, in callHandlers
    hdlr.handle(record)
  File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 854, in handle
    self.emit(record)
  File "/root/python/cobra/cobra/log.py", line 134, in emit
    self.handleError(record)
Message: '[CLI] Target directory: /root/python/cobra/tests/vulnerabilities'
Arguments: ()
--- Logging error ---

但是我直接在 centos 命令行下运行 pycharm 运行的命令是没有问题的，我是真不知道咋办了，只能打扰求助各位前辈，愿意花 20 块钱求教到底为何，聊表心意，困扰了一晚上。

[root@xiaopo-centos cobra]# /root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python /root/python/cobra/cobra.py -t /root/python/cobra/tests/vulnerabilities/
[21:05:09] [INFO] [REPORT] Report URL: ?sid=a9d7661ujlcb
[21:05:09] [INFO] [CLI] Target directory: /root/python/cobra/tests/vulnerabilities
[21:05:09] [INFO] [CLI] [STATISTIC] Language: php Framework: Spring
[21:05:09] [INFO] [CLI] [STATISTIC] Files: 15, Extensions:14, Consume: 0.0
[21:05:13] [INFO] [PUSH] 18 CVE Rules
[21:05:15] [INFO] [PUSH] 95 Rules
[21:05:18] [INFO] [SCAN] Trigger Rules/Not Trigger Rules/Off Rules: 65/2/29 Vulnerabilities (69)
5272 次点击
所在节点    Python
8 条回复
hanssx
2019-03-13 22:28:03 +08:00
pycon 群里刘师傅说了一个问题，“你用的是虚拟环境下的 python，为啥报错目录是 /usr/local/python3 呢”
我也感觉好奇怪，按道理来说，报错应该是在虚拟环境下的吧？！
chenqh
2019-03-13 22:31:23 +08:00
感觉是 log.py 文件的问题
```
def emit(self, record):
try:
message = stdout_encode(self.format(record))
stream = self.stream

if not self.is_tty:
if message and message[0] == "\r":
message = message[1:]
stream.write(message)
else:
self.output_colorized(message)
stream.write(getattr(self, 'terminator', '\n'))

self.flush()
except (KeyboardInterrupt, SystemExit):
raise
except IOError:
pass
except Exception as e:
self.handleError(record)
```
这个改成
```
def emit(self, record):
try:
message = stdout_encode(self.format(record))
stream = self.stream

if not self.is_tty:
if message and message[0] == "\r":
message = message[1:]
stream.write(message.decode(sys.stdout.encoding))
else:
self.output_colorized(message)
stream.write(getattr(self, 'terminator', '\n'))

self.flush()
except (KeyboardInterrupt, SystemExit):
raise
except IOError:
pass
except Exception as e:
self.handleError(record)
```
感觉是 pycharm log 走的
```
if not self.is_tty:
if message and message[0] == "\r":
message = message[1:]
stream.write(message.decode(sys.stdout.encoding))
```
而 terminator 走的是
```
else:
self.output_colorized(message)
```
lniwn
2019-03-13 22:36:15 +08:00
以前遇到过 pycharm 调试 asyncio，调试时下断点，pycharm 偶发退出，但是提前下好断点，再启动调试就 OK。
hanssx
2019-03-13 23:03:17 +08:00
@chenqh 多谢师傅回复，没想到确实是这样，stream.write(message)改成 stream.write(message.decode('utf-8'))就可以了。
hanssx
2019-03-13 23:09:55 +08:00
@chenqh 但是我发现在虚拟机中使用 pycharm 调试，在使用了进程池的附近也会报一样的 KeyboardInterrupt 错误，看来不是远程调试的问题，不过师傅已经帮我解决了一个问题，师傅请问微信或者 qq 私聊或者发我吧，我给师傅转钱，我的 qq 是 9614 六 2392
@lniwn 提前下好，下到出错的后面了，也不行呢，感谢师傅。
hanssx
2019-03-14 11:45:51 +08:00
最后发现好像是 py3.7 的问题，换成 py2.7 可以了。
参考： https://forums.fast.ai/t/pycharm-ide-debugger-exception/28909/4
hanssx
2019-03-14 13:29:17 +08:00
经测试，py3.6.8 亦可，大概率真是 py3.7 的问题。。
EdwardChu
2019-03-15 15:29:06 +08:00
py3.7 好像真的有些问题，我 pyltp 都装不了，只能把 python 版本改成 3.6 的才装上去了。

这是一个专为移动设备优化的页面（即为了让你能够在 Google 搜索结果里秒开这个页面），如果你希望参与 V2EX 社区的讨论，你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/544314

V2EX 是创意工作者们的社区，是一个分享自己正在做的有趣事物、交流想法，可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX