I have a redis HA with one master and two slaves. And i have around 10 million keys in my redis.
For a given flow i am deleting around 1 Million keys with a batch size of 1000 and simulatneoulsy in other flows keys are getting put into the redis.
but every time while performing delete operation i encounter java.net.SocketTimeoutException: Read timed out. I have kept timeout at 8 seconds.
Is there any issue with Redis Delete?
I am using Jedis client 2.7.
Stack Trace:
"redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out":{
"RedisInputStream.java:201":"redis.clients.util.RedisInputStream.ensureFill",
"RedisInputStream.java:40":"redis.clients.util.RedisInputStream.readByte",
"Protocol.java:141":"redis.clients.jedis.Protocol.process",
"Protocol.java:205":"redis.clients.jedis.Protocol.read",
"Connection.java:297":"redis.clients.jedis.Connection.readProtocolWithCheckingBroken",
"Connection.java:267":"redis.clients.jedis.Connection.getAll",
"Connection.java:259":"redis.clients.jedis.Connection.getAll",
Because the redis server is single thread model for process request. So all the command incoming will queued as one by one.
And about DEL operation, It not just remove the key from keyspace, it will also blocks until all the memory for the keys are free. So it slow down your redis server. you could try the UNLINK command.
This command is very similar to DEL: it removes the specified keys. Just like DEL a key is ignored if it does not exist. However the command performs the actual memory reclaiming in a different thread, so it is not blocking, while DEL is. This is where the command name comes from: the command just unlinks the keys from the keyspace. The actual removal will happen later asynchronously.
EDITED.
Maybe you should try the incrementally deleting and incrementally puting. E.g. Every minute delete 100 keys or 1000 keys.
And if your some key's type is list or set or zset which hold huge data, you might delete it delayed, first collect them, and then delete them when redis is not very busy.
if your put data flow is also this bulk load huge data into redis, incrementally putting.
Related
I have implemented storing the results of a selection from a database (list) in Redis to speed up data loading on the site. When a cms user performs any operations (creates, deletes or edits article), the keys in redis are dropped and load fresh data from the database.
But sometimes it happens that one or two users do not drop their keys after performing operations with articles and old data remains in redis. The Internet was available, nothing was turned off. Why does this happen - are there any typical reasons that I need to know?
Do I need to block the database so that there are no multiple connections? But redis seems to be single-threaded. What am I doing wrong? The function for key drop is very simple:
function articlesRedisDrop($prefix)
{
$keys = $redis->keys($prefix."*");
if(count($keys) > 0)
{
foreach($keys as $key)
{
$redis->del($key);
}
}
}
guess that an atomic question. After $redis->keys($prefix."*"), before $redis->del($key), another connection added refresh data to redis.
You can try to combine get and del operations to single lua script.
local keys = redis.call("keys",string.format("%s.*",KEYS[1]))
for _, key in pairs(keys) do
redis.call("del", key)
end
then run the script with eval command and prefix param. If you meet performance problem with keys command, you can try scan or store all prefix keys to a set then get and delete all.
I have already enabled "Service Broker" in my database. But recently I noticed that my database size is increasing day by day unexpectedly. Finally I've found out that my database size increase unexpectedly because of queue message of internal table.
The C drive is almost full, that's why I could not delete queue message from internal table. When I execute query for delete query then log file size increase accordingly, that's why I want to delete queue message without storing log.
Thanks.
While a ceph write operation , standard PUT operation - in case the data node that holds the partition (based on hash) is found dead, then does the coordinator node still sends SUCCESS ACK back for write operation ?
So the question is in case one of 3 replica nodes is found unhealthy, is the WRITE operation ACKED as failure ?
it seems it will fail in write acknowledgment in case a replica node is down if replication factor > 1 (example 2)
Data management begins with clients writing data to pools. When a client writes data to a Ceph pool, the data is sent to the primary OSD. The primary OSD commits the data locally and sends an immediate acknowledgement to the client if replication factor is 1. If the replication factor is greater than 1 (as it should be in any serious deployment) the primary OSD issues write subops to each subsidiary (secondary, tertiary, etc) OSD and awaits a response. Since we always have exactly one primary OSD, the number of subsidiary OSDs is the replication size - 1. Once all responses have arrived, depending on success, it sends acknowledgement (or failure) back to the client.
I have a remote server running Redis where I want to push a lot of a data from a Java application. Until now I used Webdis to push one command at the time which is not efficient, but I did not have any security issues because I could define the IPs that were accepted as connections and coomand authorizations while redis was not accepting requests from outside (protected mode).
I want to try to use jedis(Java API) and the implementation of pipeline for faster insertion but that means I have to open my Redis to accept requests from outside.
My question is this: Is it possible to use webdis in a similar way(pipilined mass insertion)? And if not what are the security configurations I need to make to use something like Jedis over the internet?
Thanks in advance for any answer
IMO it should be transparent for Redis driver how you set up the security. No driver or password protection will be so secure as specifically designed protocols or technologies.
In the most simple way I'd handle the security, is letting Redis listening on 127.0.0.1:<some port> and using an SSH tunnel to the machine. At least this way you can test the performance agains your current scenario.
You can also use IPSec or OpenVPN afterwards to organize private network which is able to communicate with Redis server.
This question is almost 4 years old so I hope its author has moved on by now, but in case someone else has the same issue I thought I might suggest a way to send data to Webdis more efficiently.
You can indeed make data ingest faster by batching your inserts, meaning you can use MSET to insert multiple keys in a single request (or HMSET for hashes, etc).
As an example, here's ApacheBench (ab) inserting one key 100,000 times using 100 clients:
$ ab -c 100 -n 100000 -k 'http://127.0.0.1:7379/SET/foo/bar'
[...]
Requests per second: 82235.15 [#/sec] (mean)
We're measuring 82,235 single-key inserts per second. Keep in mind that there's a lot more to HTTP benchmarking than just looking at averages (the latency distribution is still important, etc.) but this example is only about showing the difference that batching can make.
You can send commands to Webdis in one of three ways (documented here):
GET /COMMAND/arg0/.../argN
POST / with COMMAND/arg0/.../argN in the HTTP body (demonstrated below)
PUT /COMMAND/arg0.../argN-1 with argN in the HTTP body
If instead of inserting one key per request we create a file containing the MSET command to write 100 keys in a single request, we can significantly increase the write rate.
# first showing what the command looks like for 3 keys
$ echo -n 'MSET' ; for i in $(seq 1 3); do echo -n "/key-${i}/value-${i}"; done
MSET/key-1/value-1/key-2/value-2/key-3/value-3
# then saving the command to write 100 keys to a file:
$ (echo -n 'MSET' ; for i in $(seq 1 100); do echo -n "/key-${i}/value-${i}"; done) > batch-contents.txt
With this file, we can use ab to send this multi-insert file as a POST request (-p) to Webdis:
$ ab -c 100 -n 10000 -k -p ./batch-contents.txt -T 'application/x-www-form-urlencoded' 'http://127.0.0.1:7379/'
[...]
Requests per second: 18762.82 [#/sec] (mean)
This is showing 18,762 requests per second… with each request performing 100 inserts, for a total of 1,876,282 actual key inserts per second.
If you track the CPU usage of Redis while ab is running, you'll find that the MSET use case pegs it at 100% CPU while sending individual SET does not.
Once again keep in mind that this is a rough benchmark, just enough to show that there is a significant difference when you batch inserts. This is true regardless of whether Webdis is used, by the way: batching inserts from a client connecting directly to Redis should also be much faster than individual inserts.
Note: (I am the author of Webdis)
keys set with a expire is not getting cleared after expire time. For example, in reds-cli
> set hi bye
>expire hi 10
>ttl hi #=> 9
#(after 10 seconds)
>ttl hi #=> 0
>get hi #=> bye
redis version is 2.8.4. This is a master node in a sentinel setup with a single slave. Persistence is turned off. Kindly help me with debugging this issue.
If there is any data in redis with a large size, there might be problems while slave nodes sync from master nodes, which might lead to the TTL of these data in slave won't sync and so that the data in slave nodes won't get deleted.
You can use scripts to delete specific data in master node and slave nodes will delete those data whose key can't be found in master node.
Update redis.conf file to keep notify-keyspace-events Ex and then restart the redis server using redis-server /usr/local/etc/redis.conf