I try to find about custom AOF configuration. I found only that:
There are three options:
fsync every time a new command is appended to the AOF. Very very slow, very safe.
fsync every second. Fast enough (in 2.4 likely to be as fast as snapshotting), and you can lose 1 second of data if there is a disaster.
Never fsync, just put your data in the hands of the Operating System. The faster and less safe method.
Can I configure fsync which every time append a command to the AOF only for specific command (INCR)?
Is it possible ?
You could do that with a MULTI/EXEC block, i.e.:
MULTI
CONFIG SET appendfsync always
INCR somekey
CONFIG SET appendfsync no
EXEC
Related
I understand Redis AOF and RDB persistence options and have read the doc (maybe not thoroughly). What I want to ask is this: is it possible to eliminate the possibility of data loss with Redis?
Setting appendfsync to always seems to be the closest solution. However, there is stil the possibility that Redis crashes just after responding to the client with "OK" and before persisting the data to disk. There would be no way for the client to know that the data is lost, which will result in inconsistency.
As far as I'm concerned, an option to make Redis respond after fsync should resolve the issue (or maybe an additional WAITFSYNC command). Is that possible?
i think for now the safest option is to add appendonly yes in your redis config.
if you are using version 1.1 or greater one.
appendfsync always is slowest among them. if you are okay with that then sure you can use it. but if you care about your DB's performance use appendfsync everysec.
The append-only file is a fully-durable strategy for Redis. every time Redis receives a command that changes the dataset (e.g. SET) it will append it to the AOF. When you restart Redis it will re-play the AOF to rebuild the state.
details
We are using redis server in production with 6 GB data size, Initially
we thought redis can be used as memory cache only, If it restarts then we can repopulate from the persistants data store with minimal downtime.
Now we realized that re-population of data from persistence store is not a good idea at all, It is causing major service downtime.
We want to evaluate redis persistant option by using RDB and AOF combination.We tried taking RDB snapshot once in a hour and committing to the AOF file with one second interval in test environments. AOF file is growing too big in test environment only. We tried to analyze the AOF file content and noticed that lot of keys we don't want to persist to the disk, We need them only in redis memory.
Is there any way to stop logging certain type of keys (block list keys) while logging to the AOF file
Generally, Redis does not provide a way to exclude certain types of keys from persistency. If you need some keys to persist to disk and others not to, you should use two independent Redis instances - one for each type and configure their persistency settings approriately. Divide and conquer.
Note: it is possible, however, to control what gets persisted in AOF inside the context if a Lua script - see the "Selective replication of commands" section of EVAL's documentation. That said, besides the consistency risks, it would be too much of a hassle to use this approach for what you need imo.
When I write to REDIS (set up with an AOF) can I be assured that my data is persisted to the AOF correctly by the time my overall write has completed?
ie: I want to make sure my write is persistent (like a successful commit) before I continue on in my processing.
If you set appendfsync always, then every write will be persisted and flushed (OS-level fsync) before command returns. However, note that:
after OS level fsync there is a hardware level fsync, which is done by device driver and you have no control over it. If you care a lot about data safety, choose enterprise grade devices which have capacitors that allow emergency fsync in case of power failure.
appendfsync always is slow compared to other Redis modes. You can peek at numbers a bit here: aof fsync vs cassandra LSM
This question is about Redis persistence.
I'm using redis as a 'fast backend' for a social networking website. It's a single server set up. I've been transferring PostgreSQL responsibilities to Redis steadily. Currently in etc/redis/redis.conf, the appendonly setting is set to appendonly no. Snapshotting settings are save 900 1, save 300 10, save 60 10000. All this is true for production and development both. As per production logs, save 60 10000 gets invoked heavily. Does this mean that practically, I'm getting backups every 60 seconds?
Some literature suggests using AOF and RDB backups together. Thus I was weighing in on turning appendonly on and using appendfsync everysec. For anyone who has had experience of both sides of the coin:
1) Will using appendonly on and appendfsync everysec cause a performance downgrade? Will it hit the CPU? The write load is on the high side.
2) Once I restart the redis server with these new settings, I'll still lose the last 60 secs of my data, correct?
3) Are restart times something to worry about? My dump.rdb file is small; ~90MB.
I'm trying to find out more about redis persistence, and getting my expectations right. Personally, I'm fine with losing 60s of data in the case of a catastrophe, thus whether I should use AOF is also something I'm pondering. Feel free to chime in. Thanks!
Does this mean that practically, I'm getting backups every 60 seconds?
NO. Redis does a background save after 60 seconds, if there're at least 10000 keys have been changed. Otherwise, it doesn't do a background save.
Will using appendonly on and appendfsync everysec cause a performance downgrade? Will it hit the CPU? The write load is on the high side.
It depends on many things, e.g. disk performance (SSD VS HDD), write/read load (QPS), data model, and so on. You need do a benchmark with your own data in your specific environment.
Once I restart the redis server with these new settings, I'll still lose the last 60 secs of my data, correct?
NO. If you turn on both AOF and RDB, when Redis restarts, the AOF file will be used to rebuild the database. Since you config it to appendfsync everysec, you will only lose the last 1 second of data.
Are restart times something to worry about? My dump.rdb file is small; ~90MB.
If you turn on AOF, and when Redis restarts, it replays logs in AOF file to rebuild the database. Normally AOF file is larger then RDB file, and it might be slower than recovering from RDB file. Should you worry about that? Do a benchmark with your own data in your specific environment.
EDIT
IMPORTANT NOTICE
Assume that you already set Redis to use RDB saving, and write lots of data to Redis. After a while, you want to turn on AOF saving. NEVER MODIFY THE CONFIG FILE TO TURN ON AOF AND RESTART REDIS, OTHERWISE YOU'LL LOSE EVERYTHING.
Because, once you set appendonly yes in redis.conf, and restart Redis, it will load data from AOF file, no matter whether the file exists or not. If the file doesn't exist, it creates an empty file, and tries to load data from that empty file. So you'll lose everything.
In fact, you don't have to restart Redis to turn on AOF. Instead, you can use config set command to dynamically turn it on: config set appendonly yes.
As we can find in a Redis documentation (http://redis.io/topics/persistence), AOF Redis has 3 policies:
no fsync at all
fsync every second
fsync at every query
I don't want to fsync at every query, because in documentation it is said that it is very slow.
But part of my data is crucial and I cannot lose it. Is it possible to use policy where fsync is performed every second, but explicitly call fsync for given commands, to be sure that it is persisted on disk?
Minor clarification: when used with always policy, AOF is written to only after write queries.
There is no standard way or policy that does that, but you can hack this to happen by wrapping your "crucial writes" and calls to CONFIG SET in a transaction, i.e.:
SET key1 "redundant"
MULTI
CONFIG SET appendfsync always
SET key2 "crucial"
CONFIG SET appendfsync everysec
EXEC
SET key3 "transient"
I tested this in v2.8 some time ago and it appeared to work but YMMV ;)