RabbitMQ handshake timeout caused by hostname change

29 January 2018

Changing the hostname of our VM running rabbitmq-server from old-hostname to new-hostname caused RabbitMQ to timeout all handshakes. See this excerpt from /var/log/messages where a worker attempts to connect but the handshake times out exactly 10 seconds later.

Jan 25 09:38:34 new-hostname rabbitmq-server: Event crashed log handler:
Jan 25 09:38:34 new-hostname rabbitmq-server: {info_msg,<0.142.0>,
Jan 25 09:38:34 new-hostname rabbitmq-server: {<0.159.0>,"accepting AMQP connection ~p (~s)~n",
Jan 25 09:38:34 new-hostname rabbitmq-server: [<0.464.0>,"192.168.202.45:33115 -> 192.168.202.37:5672"]}}
Jan 25 09:38:34 new-hostname rabbitmq-server: function_clause
[...]
Jan 25 09:38:44 new-hostname rabbitmq-server: Event crashed log handler:
Jan 25 09:38:44 new-hostname rabbitmq-server: {error,<0.142.0>,
Jan 25 09:38:44 new-hostname rabbitmq-server: {<0.159.0>,"closing AMQP connection ~p (~s):~n~p~n",
Jan 25 09:38:44 new-hostname rabbitmq-server: [<0.464.0>,"192.168.202.45:33115 -> 192.168.202.37:5672",
Jan 25 09:38:44 new-hostname rabbitmq-server: {handshake_timeout,frame_header}]}}
Jan 25 09:38:44 new-hostname rabbitmq-server: function_clause

The 10 second timeout is presumably because of the handshake_timeout setting being set to 10000 milliseconds by default. This can be changed by editing /etc/rabbitmq/rabbitmq.config and adding {handshake_timeout, 20000} to, for example, increase the timeout to 20 seconds.

However according to the RabbitMQ documentation, an increased handshake timeout is “only necessary with very constrained clients and networks [and] in other circumstances indicate a problem elsewhere”. (The anchor tag link is wrong because the RabbitMQ documentation provides the wrong anchor tag for the “Connection Handshake Timeout” section).

As an aside, any change I made to the handshake_timeout seemed to have no effect. Even used the rabbitmqctl environment command to verify that the handshake_timeout was being read correctly.

Back to the hostname change; by default, RabbitMQ uses the machine’s hostname to determine the RabbitMQ node name. As such, changing the hostname can break the configuration of RabbitMQ. Unfortunately, the generally accepted solution seemsed to be the rather heavy-weight option of completely reinstalling RabbitMQ.

Tagged
RabbitMQ