The Redis INFO command returns a
server_load
metric, for us this is a value like 0.45. The question is what value does this represent? A percentage? A fraction of 1?
You can see from our monitoring that the load is very low:
Is 0.45 good or bad?
From https://azure.microsoft.com/en-gb/blog/investigating-timeout-exceptions-in-stackexchange-redis-for-azure-redis-cache/
"Is there a high Redis-server server load ? Using the Redis-cli client tool, you can connect to your Redis endpoint and run "INFO CPU" to check the value of server_load. A server load of 100 (maximum value) signifies that the redis server has been busy all the time (has not been idle) processing the requests.Run Slowlog from redis-cli to see if there are any requests that are taking more time to process causing server load to max out."
I think server_load only exists in Azure Redis. Haven't seen it in the standalone builds. And i'm pretty sure Microsoft would never use Unix style load indication in ... anything.
No such thing as server_load in the source code either:
https://github.com/antirez/redis/search?utf8=%E2%9C%93&q=server_load
MSOpenTech GitHub Issue link: https://github.com/MSOpenTech/redis/issues/385
Related
I try to use redis for pub/sub by two systems and one of them is now ours (other company maintain it). I would like to have time stamp when I publish something in redis channel. Can someone help with this idea?
I already user redis log with debug level of information (the highest one) - but there is no time info for pub/sub messages in the log.
I tested redis monitor: redis-cli monitor . It's exactly what I want, but it decrease the performance of the system by 50%.
The only way is to implement the time log by myself - may be SET some time info before pub command in redis? This will put in redis local time and it will be slightly before pub.
You cannot achieve the goal with pubsub.
You might want to try Redis Streaming. For each streaming message, the first part of the automatically generated ID is the Unix timestamp when the ID is generated, i.e. the message is received by Redis.
I use AWS ElastiCache Redis for our prod. I see CPU every 30 minutes of the round hour from average of 2-3% to 20%.
This is constant, which tells me it comes from schedule job.
From cloudwatch I have a suspicion it is related to KEY (and maybe SET) commands and it's latency is the only one which jumps in the same exact time as the CPU jumps.
I would like to understand what KEY (and maybe SET) commands run on a specific time, or some other way which can help me investigate this.
Thanks for any advice.
with redis-cli monitor I was able to get most of the commands running on server in a stream and get the high usage.
What are the Redis capacity requirements to support 50k consumers within one consumer group to consume and process the messages in parallel? Looking for testing an infrastructure for the same scenario and need to understand considerations.
Disclaimer: I worked in a company which used Redis in a somewhat large scale (probably less consumers than your case, but our consumers were very active), however I wasn't from the infrastructure team, but I was involved in some DevOps tasks.
I don't think you will find an exact number, so I'll try to share some tips and tricks to help you:
Be sure to read the entire Redis Admin page. There's a lot of useful information there. I'll highlight some of the tips from there:
Assuming you'll set up a Linux host, edit /etc/sysctl.conf and set a high net.core.somaxconn (RabbitMQ suggests 4096). Check the documentation of tcp-backlog config in redis.conf for an explanation about this.
Assuming you'll set up a Linux host, edit /etc/sysctl.conf and set vm.overcommit_memory = 1. Read below for a detailed explanation.
Assuming you'll set up a Linux host, edit /etc/sysctl.conf and set fs.file-max. This is very important for your use case. The Open File Handles / File Descriptors Limit is essentially the maximum number of file descriptors (each client represents a file descriptor) the SO can handle. Please check the Redis documentation on this. RabbitMQ documentation also present some useful information about it.
If you edit the /etc/sysctl.conf file, run sysctl -p to reload it.
"Make sure to disable Linux kernel feature transparent huge pages, it will affect greatly both memory usage and latency in a negative way. This is accomplished with the following command: echo never > /sys/kernel/mm/transparent_hugepage/enabled." Add this command also to /etc/rc.local to make it permanent over reboot.
In my experience Redis is not very resource-hungry, so I believe you won't have issues with CPU. Memory are directly related to how much data you intend to store in it.
If you set up a server with many cores, consider using more than one Redis Server. Redis is (mostly) single-threaded and will not use all your CPU resources if you use a single instance in a multicore environment.
Redis server also warns about wrong/risky configurations on startup (sorry for the old image):
Explanation on Overcommit Memory (vm.overcommit_memory)
Setting overcommit_memory to 1 says Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis [from Redis FAQ]
There are three possible settings for vm.overcommit_memory.
0 (zero): Check if enough memory is available and, if so, allow the allocation. If there isn’t enough memory, deny the request and return an error to the application.
1 (one): Permit memory allocation in excess of physical RAM plus swap, as defined by vm.overcommit_ratio. The vm.overcommit_ratio parameter is a
percentage added to the amount of RAM when deciding how much the kernel can overcommit. For instance, a vm.overcommit_ratio of 50 and 1 GB of
RAM would mean the kernel would permit up to 1.5 GB, plus swap, of memory to be allocated before a request failed.
2 (two): The kernel’s equivalent of "all bets are off", a setting of 2 tells the kernel to always return success to an application’s request for memory. This is absolutely as weird and scary as it sounds.
We usually use redis for caching in the Spring‘s project. My problem is that since redis is single-threaded, then our concurrent requests become serialized requests when accessing redis. then,what is the significance of using redis?
Is it only because of "It's not very frequent that CPU becomes your bottleneck with Redis, as usually Redis is either memory or network bound.
......
using pipelining Redis running on an average Linux system can deliver even 1 million requests per second......
"?
I am learning redis， Redis document FAQ
You've basically asked two questions in one question:
What is the significance of using Redis.
Well, Redis is known to be fast because it keeps the data in memory. If you ask whether being a single-threaded application is very restrictive - well, its a product, that works like this by design, maybe it could be even more performant if it was multithreaded, it depends on actual implementation under the hood after all.
In any case, it offers much more than just a "get data in memory":
- Many primitives to work with
- Configurable persistence
- Replication of data
And much more
If the question is whether the in-memory cache will be faster (you've mentioned Spring framework, so you're at Java Land) - then yes.
In fact, Spring Cache support Guava Cache (spring 5/spring boot 2 use Caffeine for the same purpose instead) - and yes it will be faster in a head-to-head comparison with Redis. But what if you have a distributed application with many instances and one instance calculated something and put it to cache, how do you get the same information from another instance without distributing the information between the instances. Well, there are tools like Hazelcast but it's out of scope for this question, the point is that when the application is beyond basic, the tasks like cache synchronization /keeping it up-to-date becomes much less obvious.
If you can deliver 1 million operations per second.
Now this question is too vague to answer:
What is the hardware that runs Redis?
What are the network configurations? (after all Redis calls are done over the network)
How often do you persist on disk (Redis has configurations for that)
Do you use replication and split the load between many Redis servers reaching an overall much faster throughput?
What commands exactly are being running under that hood?
In any case, when it comes to benchmarking you can set up your system in the option way and use the tool offered by Redis itself:
Redis Benchmarking Chapter in Redis tutorial
The tool is called redis-benchmark you can run it with various parameters and see how fast redis really is:
Here is an example (I encourage you to read the full article in the link):
$ redis-benchmark -t set,lpush -n 100000 -q
SET: 74239.05 requests per second
LPUSH: 79239.30 requests per second
This says: Connect to redis server available on localhost, run (-n) 100000 requests in a quiet mode (-q parameter) and run only tests specific for two commands: set and lpush
Logstash forwarder is light, but from logstash forwarder to logstash , there is latency over the network. [ if i am using Logstash forwarder on one machine and sending logs to Logstash which is on other machine ]
Flume /Flume-ng : CPU utilisation is high for same amount of data (for example for 2 MB ,its like 20 percent )
Fluentd : doestn't use java, its based on CRuby , but its CPU utilisation is also at peak time 30 percent, .
As per our use case we do not want to add significant load on my production boxes to just forward the log and if i use logstash i will be introducing new single point of failure so i am pretty confused to choose one among them.
Interesting performance statistics.
From my experience, logstash-forwarder is fairly light weight and encryption/compression is very helpful. This indeed might cause some latency. Is that an important factor for you? I guess latency is smaller than 2-3 seconds. I think that in many log management use cases, real-time is not a strong requirement.
At the end of the day, all these agents need to collect data from apps/files, package them and ship them over the network. This takes some cycles but in most cases, these are 2%-4% of the resources a normal server would have.
Have a look at rsyslog which has many configurations on how often it piggy backs logs. You can run it in a docker and limit resources more strictly on rsyslog or on any of the other agents (https://hub.docker.com/r/logzio/logzio-rsyslog-shipper/)
Another option would be to post logs directly from your app server with bulk HTTP post by writing your own code. It's something most open source like ELK can ingest and it something we recommend using at http://logz.io