Debugging a Python process not terminating due to active threads

16 May 2018

Situation §

A multi-threaded Python process is not terminating because a non-daemon thread is still alive. Given a running Python process, we want to identify the name of this thread.

Resolution §

Identify the PID of the Python process:

(virtualenv) [user@hostname ~]$ ps aux | grep python
root       859  0.0  0.4 553080  2288 ?        Ssl  Apr26   1:59 /usr/bin/python -Es /usr/sbin/tuned -l -P
appuser  14738  0.2  2.9 825472 14868 ?        Sl   10:11   0:02 /opt/myapp/virtualenv/bin/python -m myapp_package

Connect Pyrasite to the Python process:

(virtualenv) [user@hostname ~]$ pyrasite-shell 14738
Pyrasite Shell 2.0
Connected to '/opt/myapp/virtualenv/bin/python -m myapp_package'
Python 2.7.5 (default, Nov  6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(DistantInteractiveConsole)

>>>

Use REPL to identify extrenuous threads:

>>> import threading
>>> threading.active_count()
3

>>> threading.current_thread()
<ReversePythonShell(Thread-4, started 140499363800832)>

>>> threading.enumerate()
[<_MainThread(MainThread, stopped 140499644737344)>, <ReversePythonShell(Thread-4, started 140499363800832)>, <NaughtyThread(Thread-2, started 140498966664960)>]
>>>

In the above we identify that there are three threads: _MainThread, which is the main thread and has stopped; ReversePythonShell, which is the thread Pyrasite is running in and so is still active; and NaughtyThread, which by process of elimination is the extrenuous thread which is keeping the process alive. The name corresponds to the name of the Python class.