RabbitMQ Binary memory consumption - rabbitmq

Question

According to the images below (Rabbit 3.6.6-1) I am wondering where is all the memory being used for "Binaries" when it doesnt show the same memory usage on the "Binary references" / breakdown
Can anyone enlighten?
I suspect something needs to be "Cleaned up"... but what?
This big consumption of "Binaries" can also be seen on machines with 4 queues and no messages...
EDIT 17/07/2017:
We have found that this is mainly due to the fact that we open and close multiple connections to rabbitmq, which somehow does not seem to free up the memory in a clean way.

Answer 1

The fact that the biggest part of used memory isn't associated with any particular messages (the "binary references" part of your screenshot), suggests this memory is being used by operating system resources not directly managed by RabbitMQ. My biggest suspect would be open connections.
To test this theory, you can run netstat and see if you get similar results (assuming you're running rabbitmq on the default port - 5672):
root#rabbitmqhost:~# netstat -ntpo | grep -E ':5672\>'
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp6 0 0 xxx.xxx.xxx.xxx:5672 yyy.yyy.yyy.yyy:57656 ESTABLISHED 27872/beam.smp off (0.00/0/0)
tcp6 0 0 xxx.xxx.xxx.xxx:5672 yyy.yyy.yyy.yyy:49962 ESTABLISHED 27872/beam.smp off (0.00/0/0)
tcp6 0 0 xxx.xxx.xxx.xxx:5672 yyy.yyy.yyy.yyy:56546 ESTABLISHED 27872/beam.smp off (0.00/0/0)
tcp6 0 0 xxx.xxx.xxx.xxx:5672 yyy.yyy.yyy.yyy:50726 ESTABLISHED 27872/beam.smp off (0.00/0/0)
⋮
The interesting part is the last column showing "Timer off". This indicates this connection isn't using keepalives, which means they'll just dangle there eating up resources if the client dies without a chance of closing them gracefully.
There are two ways to avoid this problem:
TCP keepalives
application protocol heartbeats
TCP Keepalives
These are handled by the kernel. Whenever a connection sees no packages for a certain amount of time, the kernel tries to send some probes to see if the other side's still there.
Since the current Linux (e.g. 4.12) timeout defaults are pretty high (7200 seconds + 9 probes every 75 seconds > 2 hours), rabbitmq does not use them by default.
To activate them, you have to add it to your rabbitmq.config:
[
{rabbit, [
⋮
{tcp_listen_options,
[
⋮
{keepalive, true},
⋮
]
},
⋮
]},
⋮
].
and probably lower the timeouts to some more sensible values. Something like this might work (but of course YMMV):
root#rabbitmqhost:~# sysctl net.ipv4.tcp_keepalive_intvl=30
root#rabbitmqhost:~# sysctl net.ipv4.tcp_keepalive_probes=3
root#rabbitmqhost:~# sysctl net.ipv4.tcp_keepalive_time=60
Application protocol heartbeats
These are handled by the actual messaging protocols (e.g. AMQP, STOMP, MQTT), but require the client to opt-in. Since each protocol is different, you have to check the documentation to set it up in your client applications.
Conclusion
The safest option, from the perspective of avoiding dangling resources, are TCP keepalives, since you don't have to rely on your client applications behaving.
However, they are less versatile and if misconfigured on a high-throughput but "bursty" system, may lead to worse performance, since false-positives will cause reconnects.
Application protocol hearbeats are the more fine-grained option if you need to avoid this problem while also keeping your system's performance, but they require more coordination, in the sense that clients have to opt in and chose their own sensible timeouts.
Since you can never be 100% sure your clients won't die without gracefully closing connections, enabling TCP keepalives as a fallback (even with higher timeouts) might also be a good idea.