Is it possible to use redis_store output plugin for fluentd to handle huge amount of logs? - redis

Question

I'm trying to configure fluentd to send logs to redis on different server (the path is fluentbit-fluentd-redis-logstash-elastic), but I can't figure out some performance issues. If I save logs only on disc, everything is ok, all logs are saved (thousands per sec), but if I add redis_store section, the same amount of data makes fluentd much slower and the memory is still growing until next restart (day or two).
I know that it's probably because of input is faster than output, but how to handle that, how to send so many data to redis? Add more memory is not solution it can only add some time. Is it because redis can't receive data from so many threads (but redis is not overloaded, there is no queue)? I dont know if that is network issue (but in this plugin i cant try another protocol?), cpu issue (cpu is on cca 70%), or the plugin itself?
If I add redis plugin, the communication with redis is just so slow that fluentd is not fast enough and add data to memory.
Config
<system>
workers 4
root_dir /fluentd/log/buffer/
</system>
<worker 0-3>
<source>
#type forward
bind 0.0.0.0
port 9880
</source>
<label #TEST>
<match test.**>
#type forest
subtype copy
<template>
<store>
#type file
#id "test-#{worker_id}"
#log_level debug
path "fluentd/log/test-#{worker_id}.*.log"
append true
<buffer>
flush_mode interval
flush_interval 3
flush_at_shutdown true
</buffer>
<format>
#type single_value
message_key log
</format>
</store>
<store>
#type redis_store
host server_ip
port 6379
key test
store_type list
<buffer>
flush_mode interval
flush_interval 3
flush_at_shutdown true
flush_thread_count 4
</buffer>
</store>
</template>
</match>
</label>
</worker>
Any tips how to make throughput to redis better?
Thank you

Answer 1

If you are using the redistore plugin you can get much better performance by installing the hiredis gem which wraps the C redis API
https://github.com/moaikids/fluent-plugin-redisstore