Sam Hooke

These are rough notes from whatever I was working on, interested in or thinking about at the time. They vary greatly in quality and length, but prove useful to me, and hopefully to you too!

Attach shell to running Python process

pyrasite-shell (part of pyrasite) can be used to open a shell in an already running Python process.

Installation

It can be installed on CentOS 7 with the following:

$ yum install gdb
$ pip install pyrasite==2.0

Note that if you do not install gdb (as is required then pyrasite will simply hang when you launch it. Terminating pyrasite in this hung state with Ctrl-C will give the following error:

(virtualenv) [user@localhost ~]$ pyrasite-shell 4490
^CTraceback (most recent call last):
  File "/opt/myapp/virtualenv/bin/pyrasite-shell", line 11, in <module>
    sys.exit(shell())
  File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/tools/shell.py", line 30, in shell
    ipc.connect()
  File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/ipc.py", line 95, in connect
    self.wait()
  File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/ipc.py", line 151, in wait
    (clientsocket, address) = self.server_sock.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
KeyboardInterrupt

Investigation

List all processes:

$ top

There are many more running Python processes than expected:

top - 08:53:02 up 170 days, 21:20,  1 user,  load average: 29.58, 29.63, 30.00
Tasks: 148 total,  48 running,  97 sleeping,   0 stopped,   3 zombie
%Cpu(s): 88.4 us, 10.9 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
KiB Mem :   501108 total,     8084 free,   366704 used,   126320 buff/cache
KiB Swap:  1048572 total,   373276 free,   675296 used.    43508 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 4490 myuser    20   0  632496   2676    956 R  2.7  0.5  97:49.52 python
10824 myuser    20   0  632496   2728   1032 R  2.7  0.5  84:06.88 python
 3809 myuser    20   0  632240   2288    880 R  2.3  0.5  99:55.24 python
 4391 myuser    20   0  632212   2340    860 R  2.3  0.5  98:19.36 python
 4532 myuser    20   0  632496   2580    988 R  2.3  0.5  97:43.75 python
 4590 myuser    20   0  206348  14708   3752 R  2.3  2.9   0:00.17 python
 4913 myuser    20   0  632496   2532   1060 R  2.3  0.5  96:46.09 python
 5036 myuser    20   0  632496   2496   1004 R  2.3  0.5  96:25.66 python
 5127 myuser    20   0  632496   2396    908 R  2.3  0.5  96:12.94 python
 5202 myuser    20   0  632496   2276    836 R  2.3  0.5  96:02.16 python
 5221 myuser    20   0  632212   2304    852 R  2.3  0.5  96:02.98 python
 5796 myuser    20   0  632496   2516   1028 R  2.3  0.5  94:17.76 python
 5805 myuser    20   0  632496   2432    960 R  2.3  0.5  94:16.28 python
 6102 myuser    20   0  632496   2356    884 R  2.3  0.5  93:31.90 python
 6452 myuser    20   0  632496   2508   1044 R  2.3  0.5  92:31.28 python
 7392 myuser    20   0  632496   2560   1088 R  2.3  0.5  90:01.83 python
 7630 myuser    20   0  632496   2428    964 R  2.3  0.5  89:29.33 python
 8733 myuser    20   0  632484   2396    896 R  2.3  0.5  87:13.33 python
 8742 myuser    20   0  632496   2528   1068 R  2.3  0.5  87:11.00 python
 8974 myuser    20   0  632496   2348    908 R  2.3  0.5  86:47.21 python
 9598 myuser    20   0  632496   2496   1044 R  2.3  0.5  85:48.67 python
10072 myuser    20   0  632496   2492   1044 R  2.3  0.5  85:01.49 python
10164 myuser    20   0  632496   2564   1116 R  2.3  0.5  84:52.68 python
10613 myuser    20   0  632496   2464    996 R  2.3  0.5  84:20.55 python
10885 myuser    20   0  632496   2748    972 R  2.3  0.5  84:00.79 python
 3937 myuser    20   0  632496   2524   1064 R  2.0  0.5  99:37.06 python
 6318 myuser    20   0  632496   2512   1040 R  2.0  0.5  92:54.76 python
 6887 myuser    20   0  632496   2424    968 R  2.0  0.5  91:07.90 python
 7159 myuser    20   0  632496   2416    964 R  2.0  0.5  90:30.53 python
 7742 myuser    20   0  632496   2736   1040 R  2.0  0.5  89:16.37 python
 8211 myuser    20   0  632212   2024    716 R  2.0  0.4  88:17.68 python
 8420 myuser    20   0  632496   2440    968 R  2.0  0.5  87:53.67 python
 8610 myuser    20   0  632496   2452    960 R  2.0  0.5  87:26.45 python
 8751 myuser    20   0  632496   2316    856 R  2.0  0.5  87:10.06 python
 8802 myuser    20   0  632496   2596   1100 R  2.0  0.5  87:06.02 python
 8840 myuser    20   0  632496   2456    976 R  2.0  0.5  87:02.70 python
 8852 myuser    20   0  632228   2384    924 R  2.0  0.5  87:06.53 python
 9281 myuser    20   0  632212   2464   1016 R  2.0  0.5  86:23.56 python
 9879 myuser    20   0  632496   2556   1072 R  2.0  0.5  85:21.93 python
10063 myuser    20   0  632484   2600   1076 R  2.0  0.5  85:04.55 python
10091 myuser    20   0  632496   2408    976 R  2.0  0.5  85:00.01 python
10183 myuser    20   0  632496   2464   1016 R  2.0  0.5  84:49.73 python
10570 myuser    20   0  632212   2416    924 R  2.0  0.5  84:30.06 python
10594 myuser    20   0  632496   2436    960 R  2.0  0.5  84:22.37 python
10866 myuser    20   0  632228   2616    940 R  2.0  0.5  84:03.38 python
 3104 myuser    20   0  415004  84048   1436 S  1.0 16.8  20:53.12 python
25400 myuser    20   0  566692  12832   1420 S  1.0  2.6  44:53.03 python
   10 root      20   0       0      0      0 S  0.3  0.0  23:32.50 rcu_sched
   11 root      20   0       0      0      0 R  0.3  0.0  59:25.12 rcuos/0
 3059 myuser    20   0  590824  52480   1560 S  0.3 10.5  22:00.06 python
    1 root      20   0  198112   7888   1484 S  0.0  1.6 245:14.51 systemd

The VM is too slow to use effectively. Kill many of these erroneous processes:

$ pgrep -f file_that_python_process_is_executing.py | xargs kill

Better:

top - 09:03:19 up 170 days, 21:30,  1 user,  load average: 3.11, 17.91, 25.39
Tasks: 149 total,   8 running, 138 sleeping,   0 stopped,   3 zombie
%Cpu(s): 33.4 us,  7.4 sy,  0.0 ni, 58.8 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :   501108 total,     7096 free,   360592 used,   133420 buff/cache
KiB Swap:  1048572 total,   377364 free,   671208 used.    49596 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 7742 myuser    20   0  632496   2732   1036 R  6.3  0.5  89:36.12 python
 5127 myuser    20   0  632496   2392    904 R  5.6  0.5  96:32.02 python
 8420 myuser    20   0  632496   2436    964 R  5.6  0.5  88:12.51 python
10613 myuser    20   0  632496   2460    992 R  5.6  0.5  84:39.16 python
10866 myuser    20   0  632228   2612    936 R  5.6  0.5  84:23.10 python
10885 myuser    20   0  632496   2744    968 R  5.6  0.5  84:20.47 python
 3059 myuser    20   0  590824  52468   1560 S  0.7 10.5  22:03.19 python
 3104 myuser    20   0  415004  84048   1436 S  0.7 16.8  20:56.42 python
25400 myuser    20   0  566692  12828   1420 S  0.7  2.6  44:58.08 python
   10 root      20   0       0      0      0 S  0.3  0.0  23:33.18 rcu_sched
   11 root      20   0       0      0      0 R  0.3  0.0  59:26.71 rcuos/0
 4260 myuser    20   0  146284   2076   1404 R  0.3  0.4   0:02.47 top
 4490 myuser    20   0  632496   2836   1012 S  0.3  0.6  97:59.88 python
 4556 root      20   0       0      0      0 S  0.3  0.0   0:00.05 kworker/u2:1
 4595 root      20   0       0      0      0 S  0.3  0.0   0:00.37 kworker/0:0
 5202 myuser    20   0  632496   2440    892 S  0.3  0.5  96:13.90 python
 8840 myuser    20   0  632496   2616   1032 S  0.3  0.5  87:12.84 python
10091 myuser    20   0  632496   2564   1032 S  0.3  0.5  85:14.20 python
10183 myuser    20   0  632496   2628   1072 S  0.3  0.5  85:02.20 python
    1 root      20   0  198112   8024   1608 S  0.0  1.6 245:14.55 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:03.31 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0  15:28.61 ksoftirqd/0
    7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0

Attach the shell to one of these:

(virtualenv) [user@localhost ~]$ pyrasite-shell 7742
Pyrasite Shell 2.0
Connected to '/opt/myapp/virtualenv/bin/python my_script.py --arg-1=foo --arg-2=bar'
Python 2.7.5 (default, Nov  6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(DistantInteractiveConsole)

>>>
<< Previous: Hugo 0.37 does not support h6 markdown heading
Next: Removing unused COM ports from Windows >>