RabbitMQ delay my message randomly - rabbitmq

Question

here is my rabbitmqctl status：
[{pid,32074},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.2.2"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.2"},
{rabbit,"RabbitMQ","3.2.2"},
{os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.2"},
{webmachine,"webmachine","1.10.3-rmq3.2.2-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.2-git680dba8"},
{xmerl,"XML parser","1.2.10"},
{inets,"INETS CXC 138 49","5.7.1"},
{mnesia,"MNESIA CXC 138 12","4.5"},
{amqp_client,"RabbitMQ AMQP Client","3.2.2"},
{sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:32:32] [rq:32] [async-threads:30] [kernel-poll:true]\n"},
{memory,
[{total,1954308048},
{connection_procs,619048024},
{queue_procs,166111144},
{plugins,4423520},
{other_proc,46207032},
{mnesia,44407568},
{mgmt_db,331614464},
{msg_index,6694584},
{other_ets,30005328},
{binary,63825504},
{code,17629100},
{atom,6531121},
{other_system,617810659}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,53967541043},
{disk_free_limit,50000000},
{disk_free,51883839488},
{file_descriptors,
[{total_limit,655260},
{total_used,12659},
{sockets_limit,589732},
{sockets_used,12657}]},
{processes,[{limit,1048576},{used,125740}]},
{run_queue,1},
{uptime,33320350}]
My queue was empty at most of time. But there are 84440 exchanges and 8917 queues in my RabbitMQ.
My problem is the message may delay for a very long time since I send it.
I tried to get some info from the spot.Here's my monitor chart's when the delay happen.
What we can see is that overview and my exchange is fine.But my message in queue is delay in Deliver and Acknowledge.
But from my app log.
2017-08-10 17:23:08.738 4219 INFO trove.openstack.common.rpc.amqp [-] [ProxyCallback]Received Message with Timestamp:2017-08-10T17:23:08, duration:0.737949s, unique_id:a17186068cae447bbada7a0f24ff45ef
the 17:23 message was received without delay.And then I had sent ACK back to MQ:
2017-08-10 17:23:08.739 4219 DEBUG trove.openstack.common.rpc.common [-] Consume Massage with ACK True
while my 17:43 Message was delayed by 215.895117s, Really confused me.
2017-08-10 17:43:53.895 4219 INFO trove.openstack.common.rpc.amqp [-] [ProxyCallback]Received Message with Timestamp:2017-08-10T17:40:18, duration:215.895117s, unique_id:dc04b94c8fa64978bc9d681b020f4500

Answer 1

Finally, I found that is my network problem.
Sometimes the message will be drop in delivery, And RabbitMQ takes a very long time timeout and retry message delivery.
As result my message delayed.
The problem stem from security group rules, a tcp connection will be disconnected by Gateway without reset (just drop packet) after 900s no packet send.
And system wide tcp keep alive(tcp_keepalive_intvl) setup to 1200s.
So I just set tcp_keepalive_intvl with sysctl to 600s, the problem solved.