RabbitMQ lager_error_logger_h dropped messages - rabbitmq

Question

Help please solve the problem.
There are:
RabbitMQ - 3.7.2
Erlang - 20.1
Connections: 527
Channels: 500
Exchanges: 49
Queues: 4437
Consumers: 131
Publish rate ~ 200/s
Ack rate ~ 200/s
Config:
disk_free_limit.absolute = 5GB
log.default.level = warning
log.file.level = warning
In the logs constantly appear such messages:
11:42:16.000 [warning] <0.32.0> lager_error_logger_h dropped 105 messages in the last second that exceeded the limit of 100 messages/sec
11:42:17.000 [warning] <0.32.0> lager_error_logger_h dropped 101 messages in the last second that exceeded the limit of 100 messages/sec
11:42:18.000 [warning] <0.32.0> lager_error_logger_h dropped 177 messages in the last second that exceeded the limit of 100 messages/sec
How to get rid of them correctly? How to remove this messages from logs?

Answer 1

The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
The message means that RabbitMQ is generating a very large number of error messages and that they are being dropped to avoid filling the log rapidly. If "dropped X messages in the last second" is the only message you are seeing in the logs, you need to determine what the messages are that are being dropped to find the root of the problem. You can do this by temporarily raising that limit by running the following command:
rabbitmqctl eval '[lager:set_loghwm(H, 250) || H <- gen_event:which_handlers(lager_event)].'
You should then see a much larger number of messages that will reveal the underlying issue. To revert back to the previous setting, run this command:
rabbitmqctl eval '[lager:set_loghwm(H, 50) || H <- gen_event:which_handlers(lager_event)].'