Attach shell to running Python process
pyrasite-shell (part of pyrasite) can be used to open a shell in an already running Python process.
Installation §
It can be installed on CentOS 7 with the following:
$ yum install gdb
$ pip install pyrasite==2.0
Note that if you do not install gdb (as is required then pyrasite will simply hang when you launch it. Terminating pyrasite in this hung state with Ctrl-C will give the following error:
(virtualenv) [user@localhost ~]$ pyrasite-shell 4490
^CTraceback (most recent call last):
File "/opt/myapp/virtualenv/bin/pyrasite-shell", line 11, in <module>
sys.exit(shell())
File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/tools/shell.py", line 30, in shell
ipc.connect()
File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/ipc.py", line 95, in connect
self.wait()
File "/opt/myapp/virtualenv/lib/python2.7/site-packages/pyrasite/ipc.py", line 151, in wait
(clientsocket, address) = self.server_sock.accept()
File "/usr/lib64/python2.7/socket.py", line 202, in accept
sock, addr = self._sock.accept()
KeyboardInterrupt
Investigation §
List all processes:
$ top
There are many more running Python processes than expected:
top - 08:53:02 up 170 days, 21:20, 1 user, load average: 29.58, 29.63, 30.00
Tasks: 148 total, 48 running, 97 sleeping, 0 stopped, 3 zombie
%Cpu(s): 88.4 us, 10.9 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
KiB Mem : 501108 total, 8084 free, 366704 used, 126320 buff/cache
KiB Swap: 1048572 total, 373276 free, 675296 used. 43508 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4490 myuser 20 0 632496 2676 956 R 2.7 0.5 97:49.52 python
10824 myuser 20 0 632496 2728 1032 R 2.7 0.5 84:06.88 python
3809 myuser 20 0 632240 2288 880 R 2.3 0.5 99:55.24 python
4391 myuser 20 0 632212 2340 860 R 2.3 0.5 98:19.36 python
4532 myuser 20 0 632496 2580 988 R 2.3 0.5 97:43.75 python
4590 myuser 20 0 206348 14708 3752 R 2.3 2.9 0:00.17 python
4913 myuser 20 0 632496 2532 1060 R 2.3 0.5 96:46.09 python
5036 myuser 20 0 632496 2496 1004 R 2.3 0.5 96:25.66 python
5127 myuser 20 0 632496 2396 908 R 2.3 0.5 96:12.94 python
5202 myuser 20 0 632496 2276 836 R 2.3 0.5 96:02.16 python
5221 myuser 20 0 632212 2304 852 R 2.3 0.5 96:02.98 python
5796 myuser 20 0 632496 2516 1028 R 2.3 0.5 94:17.76 python
5805 myuser 20 0 632496 2432 960 R 2.3 0.5 94:16.28 python
6102 myuser 20 0 632496 2356 884 R 2.3 0.5 93:31.90 python
6452 myuser 20 0 632496 2508 1044 R 2.3 0.5 92:31.28 python
7392 myuser 20 0 632496 2560 1088 R 2.3 0.5 90:01.83 python
7630 myuser 20 0 632496 2428 964 R 2.3 0.5 89:29.33 python
8733 myuser 20 0 632484 2396 896 R 2.3 0.5 87:13.33 python
8742 myuser 20 0 632496 2528 1068 R 2.3 0.5 87:11.00 python
8974 myuser 20 0 632496 2348 908 R 2.3 0.5 86:47.21 python
9598 myuser 20 0 632496 2496 1044 R 2.3 0.5 85:48.67 python
10072 myuser 20 0 632496 2492 1044 R 2.3 0.5 85:01.49 python
10164 myuser 20 0 632496 2564 1116 R 2.3 0.5 84:52.68 python
10613 myuser 20 0 632496 2464 996 R 2.3 0.5 84:20.55 python
10885 myuser 20 0 632496 2748 972 R 2.3 0.5 84:00.79 python
3937 myuser 20 0 632496 2524 1064 R 2.0 0.5 99:37.06 python
6318 myuser 20 0 632496 2512 1040 R 2.0 0.5 92:54.76 python
6887 myuser 20 0 632496 2424 968 R 2.0 0.5 91:07.90 python
7159 myuser 20 0 632496 2416 964 R 2.0 0.5 90:30.53 python
7742 myuser 20 0 632496 2736 1040 R 2.0 0.5 89:16.37 python
8211 myuser 20 0 632212 2024 716 R 2.0 0.4 88:17.68 python
8420 myuser 20 0 632496 2440 968 R 2.0 0.5 87:53.67 python
8610 myuser 20 0 632496 2452 960 R 2.0 0.5 87:26.45 python
8751 myuser 20 0 632496 2316 856 R 2.0 0.5 87:10.06 python
8802 myuser 20 0 632496 2596 1100 R 2.0 0.5 87:06.02 python
8840 myuser 20 0 632496 2456 976 R 2.0 0.5 87:02.70 python
8852 myuser 20 0 632228 2384 924 R 2.0 0.5 87:06.53 python
9281 myuser 20 0 632212 2464 1016 R 2.0 0.5 86:23.56 python
9879 myuser 20 0 632496 2556 1072 R 2.0 0.5 85:21.93 python
10063 myuser 20 0 632484 2600 1076 R 2.0 0.5 85:04.55 python
10091 myuser 20 0 632496 2408 976 R 2.0 0.5 85:00.01 python
10183 myuser 20 0 632496 2464 1016 R 2.0 0.5 84:49.73 python
10570 myuser 20 0 632212 2416 924 R 2.0 0.5 84:30.06 python
10594 myuser 20 0 632496 2436 960 R 2.0 0.5 84:22.37 python
10866 myuser 20 0 632228 2616 940 R 2.0 0.5 84:03.38 python
3104 myuser 20 0 415004 84048 1436 S 1.0 16.8 20:53.12 python
25400 myuser 20 0 566692 12832 1420 S 1.0 2.6 44:53.03 python
10 root 20 0 0 0 0 S 0.3 0.0 23:32.50 rcu_sched
11 root 20 0 0 0 0 R 0.3 0.0 59:25.12 rcuos/0
3059 myuser 20 0 590824 52480 1560 S 0.3 10.5 22:00.06 python
1 root 20 0 198112 7888 1484 S 0.0 1.6 245:14.51 systemd
The VM is too slow to use effectively. Kill many of these erroneous processes:
$ pgrep -f file_that_python_process_is_executing.py | xargs kill
Better:
top - 09:03:19 up 170 days, 21:30, 1 user, load average: 3.11, 17.91, 25.39
Tasks: 149 total, 8 running, 138 sleeping, 0 stopped, 3 zombie
%Cpu(s): 33.4 us, 7.4 sy, 0.0 ni, 58.8 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 501108 total, 7096 free, 360592 used, 133420 buff/cache
KiB Swap: 1048572 total, 377364 free, 671208 used. 49596 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7742 myuser 20 0 632496 2732 1036 R 6.3 0.5 89:36.12 python
5127 myuser 20 0 632496 2392 904 R 5.6 0.5 96:32.02 python
8420 myuser 20 0 632496 2436 964 R 5.6 0.5 88:12.51 python
10613 myuser 20 0 632496 2460 992 R 5.6 0.5 84:39.16 python
10866 myuser 20 0 632228 2612 936 R 5.6 0.5 84:23.10 python
10885 myuser 20 0 632496 2744 968 R 5.6 0.5 84:20.47 python
3059 myuser 20 0 590824 52468 1560 S 0.7 10.5 22:03.19 python
3104 myuser 20 0 415004 84048 1436 S 0.7 16.8 20:56.42 python
25400 myuser 20 0 566692 12828 1420 S 0.7 2.6 44:58.08 python
10 root 20 0 0 0 0 S 0.3 0.0 23:33.18 rcu_sched
11 root 20 0 0 0 0 R 0.3 0.0 59:26.71 rcuos/0
4260 myuser 20 0 146284 2076 1404 R 0.3 0.4 0:02.47 top
4490 myuser 20 0 632496 2836 1012 S 0.3 0.6 97:59.88 python
4556 root 20 0 0 0 0 S 0.3 0.0 0:00.05 kworker/u2:1
4595 root 20 0 0 0 0 S 0.3 0.0 0:00.37 kworker/0:0
5202 myuser 20 0 632496 2440 892 S 0.3 0.5 96:13.90 python
8840 myuser 20 0 632496 2616 1032 S 0.3 0.5 87:12.84 python
10091 myuser 20 0 632496 2564 1032 S 0.3 0.5 85:14.20 python
10183 myuser 20 0 632496 2628 1072 S 0.3 0.5 85:02.20 python
1 root 20 0 198112 8024 1608 S 0.0 1.6 245:14.55 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:03.31 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 15:28.61 ksoftirqd/0
7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
Attach the shell to one of these:
(virtualenv) [user@localhost ~]$ pyrasite-shell 7742
Pyrasite Shell 2.0
Connected to '/opt/myapp/virtualenv/bin/python my_script.py --arg-1=foo --arg-2=bar'
Python 2.7.5 (default, Nov 6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(DistantInteractiveConsole)
>>>