I used redis for synchronize some datas.
Precondition : Data inserted into redis continuously. (About 30,000 in 10 minutes)
Here is work flow that execute every 5 minutes.
Scan keys by specific pattern(ex. 'users*')
Get all values by keys
Flush all keys
In workflow 1, I used scan_iter() to avoid locking.
I wonder that in my workflow, there is any things to causing redis lock?
If data insertion and scanning keys occur simultaneously, it can cause locking?
If you are not using the ASYNC option then FLUSHDB and FLUSHALL are blocking commands.
https://redis.io/commands/flushall
Related
Our use case is using Redis Cache in our main flow to store centralized data for our processing units in order to improve performance.
In order to persist the data from Redis to the DB we came up with a few approaches:
Writing the data to the DB right when we writing it to Redis (this approach isn't practical as it impacts the performance)
Having a duplicate entry inserted with TTL and listening to Redis events in a different process which in turn will insert them to the DB. (downside is we're losing events in case of this process being down, we can scale out the number of processes for availability in that case)
Having a Redis SortedSet holding the last update time of the entry and ZRange in a different process periodically to query for the last keys to be updated and insert them into the DB. (downside is inserting to the SortedSet is log2n and impacting performance in the main flow)
Is there another approach we're missing? We're using Azure Redis for Cache.
Thanks.
I am using redis to store some pretty large bitsets. Redis is run in master/slave sentinel mode.
I got curious about the replication performance for very big bitsets (my bitset has a size of +-100Kbyte).
From the documentation: Async replication works by sending a stream of commands between master and slave.
Can I expect those commands to update a single bit in a slave or do they copy entire keys each time? Obviously I would prefer SETBIT commands to be passed instead of setting entire keys in order to decrease network traffic.
Async replication will only pass the write command eg SETBIT to the replica in most cases.
If the replica falls too far behind however, the replica will get flushed (cleared out) and a full resync will occur. This happens if there is a lot of latency and if there are a large number of writes. If you see this happening you can tune your replication buffers to lower the possibility of a full sync
I have a key (hash type) in redis
Key is
service_status:cluster_1
Value is like below
{
service_1: normal,
service_2: normal,
service_3: normal,
service_4: normal,
service_5: down
...
}
The system is a monitor system. This data is used to store the services status of one cluster.
There are thousands of services in the cluster, so thousands of update request may hit redis to update the same key at the same time.
My concern is how redis handle this? Will there be some lock since these update pointing to the same data?
Redis is single-threaded so there are no "parallel" updates, and therefore no need for locking. Operations in general and updates to a specific hash key, in particular, are executed one at a time.
Recently, I met a problem when I use setbit in redis. As I use redis as a bloomFilter part for store, 0.2 billion data cost 380MB memory for 99.99% accuracy. Every day I need to delete the redis key for bloomfilter and create a new one, but found slow log, and this may affect other service in product environment. Counld anybody give a better suggest what to do to forbid this? thx a lot~
according command costs(ms):
DEL bloomFilterKey
use(microseconds)：83886
Freeing the large mount of memory, i.e. 380MB, costs too much time, and blocks Redis.
In order to avoid this, you can upgrade your Redis to version 4.0, and use the new command UNLINK to delete the key. This command frees memory in a different thread, and won't block Redis.
I use redis "flushdb" to flush all data in Redis but caused redis-server went away, I wondered the problem cound be cleaning a large amount of keys. So is there any idea for flushing Redis in a smoothly way? Maybe with more time for flushing all data?
flushall is "delete all keys" as described here: http://redis.io/commands/flushall
Delete operations are blocking operations.
Large delete operations may block redis 1minute or more. (e.g. you delete a 16GB hash with tons of keys)
You should write a Script which uses cursors to do this.
//edit:
I found my old answer here and wanted to be more specific providing resources:
Large number of keys, use SCAN to iterate over them with a cursor and do a gracefully cleanup in smaller batches.
Large hash, use either UNLINK command to async delete or HSCAN to iterate over it with a cursor and do a gracefully cleanup.