Sam Hooke

Attach shell to running Python process

pyrasite-shell (part of pyrasite) can be used to open a shell in an already running Python process.

Installation §

It can be installed on CentOS 7 with the following:

$ yum install gdb
$ pip install pyrasite==2.0

Note that if you do not install gdb (as is required then pyrasite will simply hang when you launch it. Terminating pyrasite in this hung state with Ctrl-C will give the following error:

(virtualenv) [user@localhost ~]$ pyrasite-shell 4490
^CTraceback (most recent call last):
  File "/opt/myapp/virtualenv/bin/pyrasite-shell", line 11, in <module>
    sys.exit(shell())
  File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/tools/shell.py", line 30, in shell
    ipc.connect()
  File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/ipc.py", line 95, in connect
    self.wait()
  File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/ipc.py", line 151, in wait
    (clientsocket, address) = self.server_sock.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
KeyboardInterrupt

Investigation §

List all processes:

$ top

There are many more running Python processes than expected:

top - 08:53:02 up 170 days, 21:20,  1 user,  load average: 29.58, 29.63, 30.00
Tasks: 148 total,  48 running,  97 sleeping,   0 stopped,   3 zombie
%Cpu(s): 88.4 us, 10.9 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem :   501108 total,     8084 free,   366704 used,   126320 buff/cache
KiB Swap:  1048572 total,   373276 free,   675296 used.    43508 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 4490 myuser    20   0  632496   2676    956 R  2.7  0.5  97:49.52 python
10824 myuser    20   0  632496   2728   1032 R  2.7  0.5  84:06.88 python
 3809 myuser    20   0  632240   2288    880 R  2.3  0.5  99:55.24 python
 4391 myuser    20   0  632212   2340    860 R  2.3  0.5  98:19.36 python
 4532 myuser    20   0  632496   2580    988 R  2.3  0.5  97:43.75 python
 4590 myuser    20   0  206348  14708   3752 R  2.3  2.9   0:00.17 python
 4913 myuser    20   0  632496   2532   1060 R  2.3  0.5  96:46.09 python
 5036 myuser    20   0  632496   2496   1004 R  2.3  0.5  96:25.66 python
 5127 myuser    20   0  632496   2396    908 R  2.3  0.5  96:12.94 python
 5202 myuser    20   0  632496   2276    836 R  2.3  0.5  96:02.16 python
 5221 myuser    20   0  632212   2304    852 R  2.3  0.5  96:02.98 python
 5796 myuser    20   0  632496   2516   1028 R  2.3  0.5  94:17.76 python
 5805 myuser    20   0  632496   2432    960 R  2.3  0.5  94:16.28 python
 6102 myuser    20   0  632496   2356    884 R  2.3  0.5  93:31.90 python
 6452 myuser    20   0  632496   2508   1044 R  2.3  0.5  92:31.28 python
 7392 myuser    20   0  632496   2560   1088 R  2.3  0.5  90:01.83 python
 7630 myuser    20   0  632496   2428    964 R  2.3  0.5  89:29.33 python
 8733 myuser    20   0  632484   2396    896 R  2.3  0.5  87:13.33 python
 8742 myuser    20   0  632496   2528   1068 R  2.3  0.5  87:11.00 python
 8974 myuser    20   0  632496   2348    908 R  2.3  0.5  86:47.21 python
 9598 myuser    20   0  632496   2496   1044 R  2.3  0.5  85:48.67 python
10072 myuser    20   0  632496   2492   1044 R  2.3  0.5  85:01.49 python
10164 myuser    20   0  632496   2564   1116 R  2.3  0.5  84:52.68 python
10613 myuser    20   0  632496   2464    996 R  2.3  0.5  84:20.55 python
10885 myuser    20   0  632496   2748    972 R  2.3  0.5  84:00.79 python
 3937 myuser    20   0  632496   2524   1064 R  2.0  0.5  99:37.06 python
 6318 myuser    20   0  632496   2512   1040 R  2.0  0.5  92:54.76 python
 6887 myuser    20   0  632496   2424    968 R  2.0  0.5  91:07.90 python
 7159 myuser    20   0  632496   2416    964 R  2.0  0.5  90:30.53 python
 7742 myuser    20   0  632496   2736   1040 R  2.0  0.5  89:16.37 python
 8211 myuser    20   0  632212   2024    716 R  2.0  0.4  88:17.68 python
 8420 myuser    20   0  632496   2440    968 R  2.0  0.5  87:53.67 python
 8610 myuser    20   0  632496   2452    960 R  2.0  0.5  87:26.45 python
 8751 myuser    20   0  632496   2316    856 R  2.0  0.5  87:10.06 python
 8802 myuser    20   0  632496   2596   1100 R  2.0  0.5  87:06.02 python
 8840 myuser    20   0  632496   2456    976 R  2.0  0.5  87:02.70 python
 8852 myuser    20   0  632228   2384    924 R  2.0  0.5  87:06.53 python
 9281 myuser    20   0  632212   2464   1016 R  2.0  0.5  86:23.56 python
 9879 myuser    20   0  632496   2556   1072 R  2.0  0.5  85:21.93 python
10063 myuser    20   0  632484   2600   1076 R  2.0  0.5  85:04.55 python
10091 myuser    20   0  632496   2408    976 R  2.0  0.5  85:00.01 python
10183 myuser    20   0  632496   2464   1016 R  2.0  0.5  84:49.73 python
10570 myuser    20   0  632212   2416    924 R  2.0  0.5  84:30.06 python
10594 myuser    20   0  632496   2436    960 R  2.0  0.5  84:22.37 python
10866 myuser    20   0  632228   2616    940 R  2.0  0.5  84:03.38 python
 3104 myuser    20   0  415004  84048   1436 S  1.0 16.8  20:53.12 python
25400 myuser    20   0  566692  12832   1420 S  1.0  2.6  44:53.03 python
   10 root      20   0       0      0      0 S  0.3  0.0  23:32.50 rcu_sched
   11 root      20   0       0      0      0 R  0.3  0.0  59:25.12 rcuos/0
 3059 myuser    20   0  590824  52480   1560 S  0.3 10.5  22:00.06 python
    1 root      20   0  198112   7888   1484 S  0.0  1.6 245:14.51 systemd

The VM is too slow to use effectively. Kill many of these erroneous processes:

$ pgrep -f file_that_python_process_is_executing.py | xargs kill

Better:

top - 09:03:19 up 170 days, 21:30,  1 user,  load average: 3.11, 17.91, 25.39
Tasks: 149 total,   8 running, 138 sleeping,   0 stopped,   3 zombie
%Cpu(s): 33.4 us,  7.4 sy,  0.0 ni, 58.8 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :   501108 total,     7096 free,   360592 used,   133420 buff/cache
KiB Swap:  1048572 total,   377364 free,   671208 used.    49596 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 7742 myuser    20   0  632496   2732   1036 R  6.3  0.5  89:36.12 python
 5127 myuser    20   0  632496   2392    904 R  5.6  0.5  96:32.02 python
 8420 myuser    20   0  632496   2436    964 R  5.6  0.5  88:12.51 python
10613 myuser    20   0  632496   2460    992 R  5.6  0.5  84:39.16 python
10866 myuser    20   0  632228   2612    936 R  5.6  0.5  84:23.10 python
10885 myuser    20   0  632496   2744    968 R  5.6  0.5  84:20.47 python
 3059 myuser    20   0  590824  52468   1560 S  0.7 10.5  22:03.19 python
 3104 myuser    20   0  415004  84048   1436 S  0.7 16.8  20:56.42 python
25400 myuser    20   0  566692  12828   1420 S  0.7  2.6  44:58.08 python
   10 root      20   0       0      0      0 S  0.3  0.0  23:33.18 rcu_sched
   11 root      20   0       0      0      0 R  0.3  0.0  59:26.71 rcuos/0
 4260 myuser    20   0  146284   2076   1404 R  0.3  0.4   0:02.47 top
 4490 myuser    20   0  632496   2836   1012 S  0.3  0.6  97:59.88 python
 4556 root      20   0       0      0      0 S  0.3  0.0   0:00.05 kworker/u2:1
 4595 root      20   0       0      0      0 S  0.3  0.0   0:00.37 kworker/0:0
 5202 myuser    20   0  632496   2440    892 S  0.3  0.5  96:13.90 python
 8840 myuser    20   0  632496   2616   1032 S  0.3  0.5  87:12.84 python
10091 myuser    20   0  632496   2564   1032 S  0.3  0.5  85:14.20 python
10183 myuser    20   0  632496   2628   1072 S  0.3  0.5  85:02.20 python
    1 root      20   0  198112   8024   1608 S  0.0  1.6 245:14.55 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:03.31 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0  15:28.61 ksoftirqd/0
    7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0

Attach the shell to one of these:

(virtualenv) [user@localhost ~]$ pyrasite-shell 7742
Pyrasite Shell 2.0
Connected to '/opt/myapp/virtualenv/bin/python my_script.py --arg-1=foo --arg-2=bar'
Python 2.7.5 (default, Nov  6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(DistantInteractiveConsole)

>>>

These are rough notes from whatever I was working on, interested in or thinking about at the time. They vary greatly in quality and length, but prove useful to me, and hopefully to you too!

← Previous: Hugo 0.37 does not support h6 markdown heading
Next: Removing unused COM ports from Windows →