V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
hanssx
V2EX  ›  Python

使用 pycharm 对 Python 项目调试遇到非常诡异的现象

  •  
  •   hanssx · 2019-03-13 21:47:12 +08:00 · 5272 次点击
    这是一个创建于 2109 天前的主题,其中的信息可能已经有所发展或是发生改变。

    调试项目: https://github.com/WhaleShark-Team/cobra

    脚本命令:python cobra.py -t /root/php/dvwa/vulnerabilities/sqli_blind/ -d

    我在公司开发机上面 pipenv 搭建使用的是 py3.7,一开始我是用的 pycharm 远程调试,Run 没有任何问题,但是 Debug 会在进程池代码 multiprocessing.Pool()附近报错 KeyboardInterrupt 吗,实际上我什么也没有做,Run 没事儿,Debug 点一下就会自动中断。

    ssh://[email protected]:22/root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python -u /root/.pycharm_helpers/pydev/pydevd.py --multiproc --qt-support --client '0.0.0.0' --port 40819 --file /root/python/cobra/cobra.py -t /root/php/dvwa/vulnerabilities/sqli_blind/ -d
    warning: Debugger speedups using cython not found. Run '"/root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python" "/root/.pycharm_helpers/pydev/setup_cython.py" build_ext --inplace' to build.
    pydev debugger: process 26949 is connecting
    
    Connected to pydev debugger (build 171.4694.94)
    [21:37:52] [DEBUG] [INIT] set logging level: debug
    [21:37:52] [DEBUG] [INIT] start scanning...
    [21:37:52] [INFO] [REPORT] Report URL: ?sid=a7df1eil4boi
    [21:37:52] [DEBUG] [PARSE-ARGS] Target Mode: folder
    [21:37:52] [DEBUG] [PARSE-ARGS] Output Mode: stream
    [21:37:52] [DEBUG] [PARSE-ARGS] target directory: /root/php/dvwa/vulnerabilities/sqli_blind/
    [21:37:52] [INFO] [CLI] Target directory: /root/php/dvwa/vulnerabilities/sqli_blind
    [21:37:52] [DEBUG] [PICKUP] /root/php/dvwa/vulnerabilities/sqli_blind
    [21:37:52] [DEBUG] [PICKUP] [FILES] |--index.php
    [21:37:52] [DEBUG] [PICKUP] [FILES] |--source
    [21:37:52] [DEBUG] [PICKUP] [FILES] |  |--medium.php
    [21:37:52] [DEBUG] [PICKUP] [FILES] |  |--impossible.php
    [21:37:52] [DEBUG] [PICKUP] [FILES] |  |--low.php
    [21:37:52] [DEBUG] [PICKUP] [FILES] |  |--high.php
    [21:37:52] [DEBUG] [PICKUP] [FILES] |--help
    [21:37:52] [DEBUG] [PICKUP] [FILES] |  |--help.php
    [21:37:52] [DEBUG] [PICKUP] [FILES] |--cookie-input.php
    [21:37:52] [DEBUG] [PICKUP] [EXTENSION-COUNT] .php : 7
    [21:37:52] [DEBUG] [DETECTION] [LANGUAGE] .php 7
    [21:37:52] [DEBUG] [DETECTION] [LANGUAGE] found the chiefly language(php), maybe have largest, continue...
    [21:37:52] [DEBUG] [DETECTION] [LANGUAGE] main language(php), tmp language(None)
    [21:37:52] [DEBUG] Dependency analysis cannot be done without finding dependency files
    [21:37:52] [INFO] [DETECTION] [FRAMEWORK] Unknown Framework
    [21:37:52] [INFO] [CLI] [STATISTIC] Language: php Framework: Unknown Framework
    [21:37:52] [INFO] [CLI] [STATISTIC] Files: 7, Extensions:1, Consume: 0.0
    [21:37:52] [DEBUG] filter dependency rules
    [21:37:52] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:53] [DEBUG] filter dependency rules
    [21:37:54] [DEBUG] filter dependency rules
    [21:37:54] [DEBUG] filter dependency rules
    [21:37:54] [DEBUG] filter dependency rules
    [21:37:54] [DEBUG] filter dependency rules
    [21:37:54] [DEBUG] filter dependency rules
    [21:37:54] [DEBUG] filter dependency rules
    [21:37:54] [INFO] [PUSH] 18 CVE Rules
    [21:37:55] [DEBUG] Dependency analysis cannot be done without finding dependency files
    [21:37:55] [DEBUG] [SCAN] [STORE] Not found vulnerabilities on this rule!
    ^CTraceback (most recent call last):
      File "/root/.pycharm_helpers/pydev/pydevd.py", line 1591, in <module>
        globals = debugger.run(setup['file'], None, None, is_module)
      File "/root/.pycharm_helpers/pydev/pydevd.py", line 1018, in run
        pydev_imports.execfile(file, globals, locals)  # execute the script
      File "/root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
        exec(compile(contents+"\n", file, 'exec'), glob, loc)
      File "/root/python/cobra/cobra.py", line 22, in <module>
        sys.exit(main())
      File "/root/python/cobra/cobra/__init__.py", line 110, in main
        cli.start(args.target, args.format, args.output, args.special_rules, a_sid, args.dels)
      File "/root/python/cobra/cobra/cli.py", line 94, in start
        language=main_language, framework=main_framework, file_count=file_count, extension_count=len(files))
      File "/root/python/cobra/cobra/engine.py", line 160, in scan
        cve_vuls = scan_cve(target_directory)
      File "/root/python/cobra/cobra/cve.py", line 406, in scan_cve
        pool.join()
      File "/usr/local/python3/lib/python3.7/multiprocessing/pool.py", line 560, in join
        p.join()
      File "/usr/local/python3/lib/python3.7/multiprocessing/process.py", line 140, in join
        res = self._popen.wait(timeout)
      File "/usr/local/python3/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
        return self.poll(os.WNOHANG if timeout == 0.0 else 0)
      File "/usr/local/python3/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
        pid, sts = os.waitpid(self.pid, flag)
    KeyboardInterrupt
    
    Process finished with exit code 1
    

    因为看不懂报错代码,所以我就使用了我搭建在 vmware 中的虚拟机 centos7,然后同样使用 pipenv 安装了 py3.7 的虚拟环境,但是诡异得是 Run 都会产生问题了,而且报错也和之前远程调试 Debug 不一样,

    /root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python /root/python/cobra/cobra.py -t /root/python/cobra/tests/vulnerabilities/
    --- Logging error ---
    Traceback (most recent call last):
      File "/root/python/cobra/cobra/log.py", line 123, in emit
        stream.write(message)
    TypeError: write() argument must be str, not bytes
    Call stack:
      File "/root/python/cobra/cobra.py", line 22, in <module>
        sys.exit(main())
      File "/root/python/cobra/cobra/__init__.py", line 110, in main
        cli.start(args.target, args.format, args.output, args.special_rules, a_sid, args.dels)
      File "/root/python/cobra/cobra/cli.py", line 64, in start
        logger.info('[REPORT] Report URL: {u}'.format(u=report))
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1332, in info
        self._log(INFO, msg, args, **kwargs)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1468, in _log
        self.handle(record)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1478, in handle
        self.callHandlers(record)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1540, in callHandlers
        hdlr.handle(record)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 854, in handle
        self.emit(record)
      File "/root/python/cobra/cobra/log.py", line 134, in emit
        self.handleError(record)
    Message: '[REPORT] Report URL: ?sid=a9d766qddopy'
    Arguments: ()
    --- Logging error ---
    Traceback (most recent call last):
      File "/root/python/cobra/cobra/log.py", line 123, in emit
        stream.write(message)
    TypeError: write() argument must be str, not bytes
    Call stack:
      File "/root/python/cobra/cobra.py", line 22, in <module>
        sys.exit(main())
      File "/root/python/cobra/cobra/__init__.py", line 110, in main
        cli.start(args.target, args.format, args.output, args.special_rules, a_sid, args.dels)
      File "/root/python/cobra/cobra/cli.py", line 75, in start
        logger.info('[CLI] Target directory: {d}'.format(d=target_directory))
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1332, in info
        self._log(INFO, msg, args, **kwargs)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1468, in _log
        self.handle(record)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1478, in handle
        self.callHandlers(record)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 1540, in callHandlers
        hdlr.handle(record)
      File "/usr/local/python3/lib/python3.7/logging/__init__.py", line 854, in handle
        self.emit(record)
      File "/root/python/cobra/cobra/log.py", line 134, in emit
        self.handleError(record)
    Message: '[CLI] Target directory: /root/python/cobra/tests/vulnerabilities'
    Arguments: ()
    --- Logging error ---
    

    但是我直接在 centos 命令行下运行 pycharm 运行的命令是没有问题的,我是真不知道咋办了,只能打扰求助各位前辈,愿意花 20 块钱求教到底为何,聊表心意,困扰了一晚上。

    [root@xiaopo-centos cobra]# /root/.local/share/virtualenvs/cobra-CV_8c13e/bin/python /root/python/cobra/cobra.py -t /root/python/cobra/tests/vulnerabilities/
    [21:05:09] [INFO] [REPORT] Report URL: ?sid=a9d7661ujlcb
    [21:05:09] [INFO] [CLI] Target directory: /root/python/cobra/tests/vulnerabilities
    [21:05:09] [INFO] [CLI] [STATISTIC] Language: php Framework: Spring
    [21:05:09] [INFO] [CLI] [STATISTIC] Files: 15, Extensions:14, Consume: 0.0
    [21:05:13] [INFO] [PUSH] 18 CVE Rules
    [21:05:15] [INFO] [PUSH] 95 Rules
    [21:05:18] [INFO] [SCAN] Trigger Rules/Not Trigger Rules/Off Rules: 65/2/29 Vulnerabilities (69)
    
    8 条回复    2019-03-15 15:29:06 +08:00
    hanssx
        1
    hanssx  
    OP
       2019-03-13 22:28:03 +08:00
    pycon 群里刘师傅说了一个问题,“你用的是虚拟环境下的 python,为啥报错目录是 /usr/local/python3 呢”
    我也感觉好奇怪,按道理来说,报错应该是在虚拟环境下的吧?!
    chenqh
        2
    chenqh  
       2019-03-13 22:31:23 +08:00   ❤️ 2
    感觉是 log.py 文件的问题
    ```
    def emit(self, record):
    try:
    message = stdout_encode(self.format(record))
    stream = self.stream

    if not self.is_tty:
    if message and message[0] == "\r":
    message = message[1:]
    stream.write(message)
    else:
    self.output_colorized(message)
    stream.write(getattr(self, 'terminator', '\n'))

    self.flush()
    except (KeyboardInterrupt, SystemExit):
    raise
    except IOError:
    pass
    except Exception as e:
    self.handleError(record)
    ```
    这个改成
    ```
    def emit(self, record):
    try:
    message = stdout_encode(self.format(record))
    stream = self.stream

    if not self.is_tty:
    if message and message[0] == "\r":
    message = message[1:]
    stream.write(message.decode(sys.stdout.encoding))
    else:
    self.output_colorized(message)
    stream.write(getattr(self, 'terminator', '\n'))

    self.flush()
    except (KeyboardInterrupt, SystemExit):
    raise
    except IOError:
    pass
    except Exception as e:
    self.handleError(record)
    ```
    感觉是 pycharm log 走的
    ```
    if not self.is_tty:
    if message and message[0] == "\r":
    message = message[1:]
    stream.write(message.decode(sys.stdout.encoding))
    ```
    而 terminator 走的是
    ```
    else:
    self.output_colorized(message)
    ```
    lniwn
        3
    lniwn  
       2019-03-13 22:36:15 +08:00 via iPhone
    以前遇到过 pycharm 调试 asyncio,调试时下断点,pycharm 偶发退出,但是提前下好断点,再启动调试就 OK。
    hanssx
        4
    hanssx  
    OP
       2019-03-13 23:03:17 +08:00
    @chenqh 多谢师傅回复,没想到确实是这样,stream.write(message)改成 stream.write(message.decode('utf-8'))就可以了。
    hanssx
        5
    hanssx  
    OP
       2019-03-13 23:09:55 +08:00
    @chenqh 但是我发现在虚拟机中使用 pycharm 调试,在使用了进程池的附近也会报一样的 KeyboardInterrupt 错误,看来不是远程调试的问题,不过师傅已经帮我解决了一个问题,师傅请问微信或者 qq 私聊或者发我吧,我给师傅转钱,我的 qq 是 9614 六 2392
    @lniwn 提前下好,下到出错的后面了,也不行呢,感谢师傅。
    hanssx
        6
    hanssx  
    OP
       2019-03-14 11:45:51 +08:00
    最后发现好像是 py3.7 的问题,换成 py2.7 可以了。
    参考: https://forums.fast.ai/t/pycharm-ide-debugger-exception/28909/4
    hanssx
        7
    hanssx  
    OP
       2019-03-14 13:29:17 +08:00
    经测试,py3.6.8 亦可,大概率真是 py3.7 的问题。。
    EdwardChu
        8
    EdwardChu  
       2019-03-15 15:29:06 +08:00
    py3.7 好像真的有些问题,我 pyltp 都装不了,只能把 python 版本改成 3.6 的才装上去了。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   2560 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 04:49 · PVG 12:49 · LAX 20:49 · JFK 23:49
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.