Python program using Stomp protocol to connect to ActiveMQ keeps disconnecting - activemq

Question

Below is the connection parameters in the python program to connect to ActiveMQ
broker_url = config_params.items('BROKERS')
conn = stomp.Connection12(broker_url,
reconnect_sleep_initial=20.0,
reconnect_sleep_increase=2.0,
reconnect_attempts_max=10,
heartbeats=(60000,60000)
)
So the ReadCheckInterval and WriteCheckInterval are set to 1 minute for the connection. It looks like the heartbeats are being missed. I am just trying to figure out if the heart beats are missing from the client or the ActiveMQ server end. Can someone help me?
Below are the logs from the Python program:
2020-02-25 12:27:16,141 - INFO - Attempting connection to host
2020-02-25 12:27:16,142 - INFO - Established connection to host
2020-02-25 12:27:16,142 - INFO - Starting receiver loop
2020-02-25 12:27:16,143 - DEBUG - Sending frame: ['STOMP', '\n', 'accept-version:1.2\n', 'client-id, 'heart-beat:60000,60000\n',]
2020-02-25 12:27:16,143 - DEBUG - Received frame: 'CONNECTED', headers={'server': 'ActiveMQ/5.15.2', 'heart-beat': '60000,60000']body=''
2020-02-25 12:27:16,143 - DEBUG - Sending frame: ['SUBSCRIBE', '\n', 'ack:auto\n', 'activemq.subscriptionName:subscriber\n']
2020-02-25 12:30:16,144 - DEBUG - Received frame: 'heartbeat', headers={}, body=None
2020-02-25 12:30:16,145 - ERROR - disconnected from broker, will attempt to reconnect...
2020-02-25 12:30:16,145 - INFO - Receiver loop ended
2020-02-25 12:30:16,320 - INFO - Attempting connection to host
2020-02-25 12:30:16,321 - INFO - Established connection to host
2020-02-25 12:30:16,321 - INFO - Starting receiver loop
2020-02-25 12:30:16,321 - DEBUG - Sending frame: ['STOMP', '\n', 'accept-version:1.2\n', 'client-id:\n', 'heart-beat:60000,60000\n']
2020-02-25 12:30:16,322 - DEBUG - Received frame: 'CONNECTED', headers={'server': 'ActiveMQ/5.15.2', 'heart-beat': '60000,60000']body=''
2020-02-25 12:30:16,322 - DEBUG - Sending frame: ['SUBSCRIBE', '\n', 'ack:auto\n', 'activemq.subscriptionName:subscriber]
I see client and the server both missing sending heart beats to each other. Below is a log where the client has missed sending the heartbeat. The connection gets established at 12:03:32. The client sends the first heart beat at 12:03:32 and then subscribes to the ActiveMQ destination. It keeps getting messages, so there is activity, until 12:12:08. Then a period of inactivity until 12:13:32 (>60 seconds) and the connection gets terminated. Is this a problem of the ActiveMQ server being too less tolerant to missed heart beats from the client. Would increasing the heartbeat interval from the client to 120 seconds help in this case?
2020-02-26 12:03:32,498 - INFO - Established connection to host, port 61613
2020-02-26 12:03:32,499 - INFO - Sending frame: 'STOMP', headers={'heart-beat': '60000,60000'}
2020-02-26 12:03:32,512 - INFO - Received frame: 'CONNECTED', headers={'heart-beat': '60000,60000'}
2020-02-26 12:03:32,513 - INFO - Sending frame: 'SUBSCRIBE'
2020-02-26 12:04:27,924 - INFO - Received frame: 'MESSAGE'
.
.
2020-02-26 12:12:08,475 - INFO - Received frame: 'MESSAGE'
2020-02-26 12:13:32,519 - INFO - Received frame: 'heartbeat'
2020-02-26 12:13:32,548 - ERROR - disconnected from broker
I also see problems os the server missing to send the heartbeat and the client getting a heartbeat timeout error. I am thinking of disabling heartbeats from the server by setting the heartbeat configuration to (120000,0). Any suggestions?

Answer 1

After some testing it turned out that even a few milliseconds delay in the client heartbeat was causing the connection to be closed by the broker.
For the same reason, from ActiveMQ server version 5.9.0,
transport.hbGracePeriodMultiplier (default=1) was added. This would increase the heartbeat timeout by a factor of the set value. Below is the JIRA link which was used to implement this feature.
https://issues.apache.org/jira/browse/AMQ-4674
I've also removed the broker heartbeat, by setting the heartbeat as (60000,0) as it was redundant.

Answer 2

The first value in the Connect hear-beat header is the 'will send' value for client heart beats to the broker. The client should be attempting to maintain a consistent heart beat at the level indicated which is defined as
smallest number of milliseconds between heart-beats that it can
guarantee
The broker will allow for some grace period based on that value after which if the client has not sent a heart beat or any other frame the connection will be closed. From the trace given the client is not sending any heart beats or other wire level activity so the broker is dropping the connection.