We're running Redis 5.0.3 on docker, with both saving and AOF turned off:
127.0.0.1:6379> config get save
1) "save"
2) ""
127.0.0.1:6379> config get appendonly
1) "appendonly"
2) "no"
Everything runs fine (no backups in the logs), until this morning when we got several DB backup logs in quick succession:
21 Mar 2019 04:12:58.453 * DB saved on disk
21 Mar 2019 04:12:58.454 * DB saved on disk
21 Mar 2019 04:12:58.456 * DB saved on disk
21 Mar 2019 04:13:50.153 * DB saved on disk
21 Mar 2019 04:13:51.573 * DB saved on disk
21 Mar 2019 04:13:52.282 * DB saved on disk
21 Mar 2019 04:21:18.539 * DB saved on disk
21 Mar 2019 04:21:18.540 * DB saved on disk
21 Mar 2019 04:21:18.541 * DB saved on disk
During this time period, Redis drops all of our keys - twice!
Any ideas why this is happening? The system is not under memory or CPU pressure, all graphs look normal.
Other useful things:
Memory usage of redis is increasing but still well within the bounds of the box (as expected as we're storing streams of data)
Number of keys is flat during this time period, until they all get dropped
Latency is flat the whole time
Redis reports no expired or evicted keys
The slow log bumps up during that time period and then is flat again immediately after.
EDIT
On further debugging using info commandstats it seems that several flushall commands were made during this time period, which would explain the DB saves from looking at source.
I have no idea why these flushes are occurring - we do not have any flush commands in our applications. Debugging continues.
Related
I've inherited a custom built webpage which uses redis server and i noticed that in every 3-5 minutes redis server peaks and uses 100% cpu for maybe 2-3 minutes.
Anyone have any ideas or clues on what i can do to optimize this?
Log file:
2276:M 23 Apr 2019 18:22:44.060 * 10 changes in 300 seconds. Saving...
2276:M 23 Apr 2019 18:22:44.356 * Background saving started by pid 16081
16081:C 23 Apr 2019 18:25:03.575 * DB saved on disk
16081:C 23 Apr 2019 18:25:03.783 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:25:04.174 * Background saving terminated with success
2276:M 23 Apr 2019 18:30:05.089 * 10 changes in 300 seconds. Saving...
2276:M 23 Apr 2019 18:30:05.396 * Background saving started by pid 16984
16984:C 23 Apr 2019 18:32:26.841 * DB saved on disk
16984:C 23 Apr 2019 18:32:27.126 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:32:27.523 * Background saving terminated with success
2276:M 23 Apr 2019 18:47:28.032 * 1 changes in 900 seconds. Saving...
2276:M 23 Apr 2019 18:47:28.334 * Background saving started by pid 18748
18748:C 23 Apr 2019 18:49:53.540 * DB saved on disk
18748:C 23 Apr 2019 18:49:53.744 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:49:54.157 * Background saving terminated with success
2276:M 23 Apr 2019 18:54:55.023 * 10 changes in 300 seconds. Saving...
2276:M 23 Apr 2019 18:54:55.328 * Background saving started by pid 19422
19422:C 23 Apr 2019 18:57:18.455 * DB saved on disk
19422:C 23 Apr 2019 18:57:18.592 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:57:18.823 * Background saving terminated with success
That's just how Redis is. The backup process involves heavy amounts of compression which will, for a time, pin a single core.
You can switch to using the AOF (append-only-file) exclusively, but that does require periodic pruning and compression as well, it's just less frequent and demanding.
The manual has some specifics, including this caveat:
There are many users using AOF alone, but we discourage it since to have an RDB snapshot from time to time is a great idea for doing database backups, for faster restarts, and in the event of bugs in the AOF engine.
I've been using Redis on a windows server for last 10 months without any issue but this morning I checked my website and saw that it's completely empty!!!
After a few minutes of investigation I realised that Redis database was empty???
Luckily I use redis as a caching solution so I still have all data in MS SQL database and I've managed to recover content of my website.
But I realised that redis has stopped saving data into dump.rdb. The last time file was updated 20.11.2015 at 11:35.
Redis config file has set
save 900 1
save 300 10
save 60 10000
and by just reloading all from MS SQL this morning I had more than 15.000 writes. So the file should be updated, right?
I run redis-check-dump dump.rdb and as result got:
Processed 7924 valid opcodes
I even run manually SAVE command and as result got:
OK <2.12>
But the file size and update date of dump.rdb is the same 20.11.2015
I just want to highlight that between 20.11.2015 and today I haven't changed anything in redis configuration or restarted the server
Any idea?
It's not the answer but at least I've managed to make Redis to start dumping data to disk.
Using console I set a new dbfilename name and now Redis is again dumping data data to disk.
It would be great if someone has a clue why it had stopped duping data to original dump file
Errors: Unexpected. Please try again.
Job ID: aaaaaaaaa.com:bbbbbbbbb:job_F1GjqdmJj3JZWDToxh_xav9hwsg
Start Time: 11:10am, 14 Apr 2014
End Time: 1:10pm, 14 Apr 2014
Destination Table: aaaaaaaaaaa.com:bbbbbbbbb:monte.ledger2
No idea why this is failing - some succeed, some do not...
Your import job is taking too long to process, so it is getting killed. We're reporting this as an internal error, which should be fixed to report a more user-friendly error instead.
We will also look into bumping the timeout (currently 2 hours per worker).
Workarounds for the issue include splitting your data into smaller files, or using uncompressed input files. The latter may sound surprising, but uncompressed files can be split into chunks and processed in parallel, but gzipped files cannot.
We run SQL Server 2005 and have a database that's about 100 GB (the MDF is 100GB and the LDF is 34 GB).
Our maintenance plan takes a full database back up every night. It's set up to
This backup size is usually around 95-100 GB but it all of a sudden grew to 120 GB, then 124 GB then 130 GB then back to 100 GB over 4 consecutive days.
Does anyone know what could cause this? I don't believe we added and then removed that much data in such a short period of time.
If your backup is larger than the MDF, this means you have a lot of log activity recorded too. SQL Server notes data changes that happen during a full backup and does a "mini" log backup to capture this.
I'd say that you need to change Index maintenance and backup timings
We are trying to have log shipping work between two database servers where the destination server has a system time set 24 hours earlier than the source system time. Is it possible to force the destination machine to restore the data (disregarding the transaction file stamps are 24 hours ahead).
How about a snapshot # 00:00 (or 23:59:59) daily? Then you could compare the days however you want without a mechanism such as log shipping in play. I'm a big fan of Snapshots (just be careful where you put the files on disk if it's a busy DB).
Here's the How to: http://msdn.microsoft.com/en-us/library/ms175876.aspx