Kombu "timed out" bug: connection hangs forever in Kombu 4.1.0
Kombu connects to RabbitMQ. However in Kombu 4.1.0 there is a bug where it waits forever if the broker is down rather than timing out. This affects Celery too. There is a fix in trunk but at time of writing there is no new release of Kombu which contains this fix. A workaround is to downgrade to Kombu 4.0.2.
# kombu==4.1.0
>>> app.control.ping()
# hangs forever
# kombu==4.0.2
>>> app.control.ping()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/venv/lib/python2.7/site-packages/celery/app/control.py", line 235, in ping
timeout=timeout, **kwargs)
File "/opt/venv/lib/python2.7/site-packages/celery/app/control.py", line 436, in broadcast
limit, callback, channel=channel,
File "/opt/venv/lib/python2.7/site-packages/kombu/pidbox.py", line 304, in _broadcast
chan = channel or self.connection.default_channel
File "/opt/venv/lib/python2.7/site-packages/kombu/connection.py", line 819, in default_channel
self.connection
File "/opt/venv/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
self._connection = self._establish_connection()
File "/opt/venv/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
conn = self.transport.establish_connection()
File "/opt/venv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/opt/venv/lib/python2.7/site-packages/amqp/connection.py", line 302, in connect
self.drain_events(timeout=self.connect_timeout)
File "/opt/venv/lib/python2.7/site-packages/amqp/connection.py", line 485, in drain_events
while not self.blocking_read(timeout):
File "/opt/venv/lib/python2.7/site-packages/amqp/connection.py", line 490, in blocking_read
frame = self.transport.read_frame()
File "/opt/venv/lib/python2.7/site-packages/amqp/transport.py", line 240, in read_frame
frame_header = read(7, True)
File "/opt/venv/lib/python2.7/site-packages/amqp/transport.py", line 415, in _read
s = recv(n - len(rbuf))
socket.timeout: timed out
>>>