I've inherited a custom built webpage which uses redis server and i noticed that in every 3-5 minutes redis server peaks and uses 100% cpu for maybe 2-3 minutes.
Anyone have any ideas or clues on what i can do to optimize this?
Log file:
2276:M 23 Apr 2019 18:22:44.060 * 10 changes in 300 seconds. Saving...
2276:M 23 Apr 2019 18:22:44.356 * Background saving started by pid 16081
16081:C 23 Apr 2019 18:25:03.575 * DB saved on disk
16081:C 23 Apr 2019 18:25:03.783 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:25:04.174 * Background saving terminated with success
2276:M 23 Apr 2019 18:30:05.089 * 10 changes in 300 seconds. Saving...
2276:M 23 Apr 2019 18:30:05.396 * Background saving started by pid 16984
16984:C 23 Apr 2019 18:32:26.841 * DB saved on disk
16984:C 23 Apr 2019 18:32:27.126 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:32:27.523 * Background saving terminated with success
2276:M 23 Apr 2019 18:47:28.032 * 1 changes in 900 seconds. Saving...
2276:M 23 Apr 2019 18:47:28.334 * Background saving started by pid 18748
18748:C 23 Apr 2019 18:49:53.540 * DB saved on disk
18748:C 23 Apr 2019 18:49:53.744 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:49:54.157 * Background saving terminated with success
2276:M 23 Apr 2019 18:54:55.023 * 10 changes in 300 seconds. Saving...
2276:M 23 Apr 2019 18:54:55.328 * Background saving started by pid 19422
19422:C 23 Apr 2019 18:57:18.455 * DB saved on disk
19422:C 23 Apr 2019 18:57:18.592 * RDB: 1 MB of memory used by copy-on-write
2276:M 23 Apr 2019 18:57:18.823 * Background saving terminated with success
That's just how Redis is. The backup process involves heavy amounts of compression which will, for a time, pin a single core.
You can switch to using the AOF (append-only-file) exclusively, but that does require periodic pruning and compression as well, it's just less frequent and demanding.
The manual has some specifics, including this caveat:
There are many users using AOF alone, but we discourage it since to have an RDB snapshot from time to time is a great idea for doing database backups, for faster restarts, and in the event of bugs in the AOF engine.
Related
We're running Redis 5.0.3 on docker, with both saving and AOF turned off:
127.0.0.1:6379> config get save
1) "save"
2) ""
127.0.0.1:6379> config get appendonly
1) "appendonly"
2) "no"
Everything runs fine (no backups in the logs), until this morning when we got several DB backup logs in quick succession:
21 Mar 2019 04:12:58.453 * DB saved on disk
21 Mar 2019 04:12:58.454 * DB saved on disk
21 Mar 2019 04:12:58.456 * DB saved on disk
21 Mar 2019 04:13:50.153 * DB saved on disk
21 Mar 2019 04:13:51.573 * DB saved on disk
21 Mar 2019 04:13:52.282 * DB saved on disk
21 Mar 2019 04:21:18.539 * DB saved on disk
21 Mar 2019 04:21:18.540 * DB saved on disk
21 Mar 2019 04:21:18.541 * DB saved on disk
During this time period, Redis drops all of our keys - twice!
Any ideas why this is happening? The system is not under memory or CPU pressure, all graphs look normal.
Other useful things:
Memory usage of redis is increasing but still well within the bounds of the box (as expected as we're storing streams of data)
Number of keys is flat during this time period, until they all get dropped
Latency is flat the whole time
Redis reports no expired or evicted keys
The slow log bumps up during that time period and then is flat again immediately after.
EDIT
On further debugging using info commandstats it seems that several flushall commands were made during this time period, which would explain the DB saves from looking at source.
I have no idea why these flushes are occurring - we do not have any flush commands in our applications. Debugging continues.
so i set up my php page and the cronjob, in order to create an excel file with data in it, everyday at 6PM, but it is not working as intended.
it executes the script every 2 days not daily.
0 11 * * * wget -q --spider http://example.com/UserReport_Export.php
it had actually created the file at 1 june, and 3 june, while 2 june was not there. any idea?
i also put the time 11 in the script cuz of server time, in order to match 6PM in my local time, maybe it affects anything? thank you
You can set the cron like this:
0 18 * * * wget -q --spider http://example.com/UserReport_Export.php
There are also online tools for cron jobs like this which can help out. The cron that you have set seems to be correct, if 11 is matching your local time.
it had actually created the file at 1 june, and 3 june, while 2 june was not there. any idea?
Is it possible for the script or anything else to interfere with your excel file?
A snapshot named snapshot 1 is created for a Stream on Jan 1 2017 (For example)
Later I create a stream out of snapshot 1 and do some code delivery. (On Feb 2017)
After a month, I create another stream out a snapshot 1. (March 2017)
Will the snapshot 1 have the same code as created on Jan 1 2017 ?
Will the snapshot 1 have the same code as created on Jan 1 2017 ?
Yes, that is the idea behind a snapshot: it is immutable... unless tou delete it (which is possible since RTC4.0) and recreate one with the same name.
But if you have not deleted it, a snapshot content never change.
I am using HSQLDB 2.3.0. I have a database that the following schema:
CREATE TABLE MEASUREMENT (ID INTEGER NOT NULL PRIMARY KEY IDENTITY, OBJ CLOB);
When I fill this table with test data, the LOBS file in my database grows:
ls -lath
-rw-rw-r-- 1 hsqldb hsqldb 35 May 6 16:37 msdb.log
-rw-rw-r-- 1 hsqldb hsqldb 85 May 6 16:37 msdb.properties
-rw-rw-r-- 1 hsqldb hsqldb 16 May 6 16:37 msdb.lck
drwxrwxr-x 2 hsqldb hsqldb 4.0K May 6 16:37 msdb.tmp
-rw-rw-r-- 1 hsqldb hsqldb 1.6M May 6 16:37 msdb.script
-rw-rw-r-- 1 hsqldb hsqldb 625M May 6 16:35 msdb.lobs
After running the following command:
TRUNCATE SCHEMA public AND COMMIT;
CHECKPOINT DEFRAG;
SHUTDOWN COMPACT;
The lobs file is still the same size:
-rw-rw-r-- 1 hsqldb hsqldb 84 May 6 16:44 msdb.properties
-rw-rw-r-- 1 hsqldb hsqldb 1.6M May 6 16:44 msdb.script
-rw-rw-r-- 1 hsqldb hsqldb 625M May 6 16:35 msdb.lobs
What is the best way to truncate the schema and get all the disk space back?
I have an application with the same problem using hsqldb 2.3.3. The .lobs file seems to be growing indefinatly even after calling "checkpoint defrag". My scenario is that I'm inserting a 1000 blobs of 300 bytes each. I'm periodically deleting them all and inserting a 1000 new blobs about the same size. After a number of rounds of this my .lobs file is now 1,3GB in size but it is really just storing around 300kB of data. Inspite of calling checkpoint defrag the .lobs file just grows and grows. Is this behavoiur a bug?
The database engine is designed for continuous use in real applications. If you have an application that uses lobs and deletes some of them, the space will be reused for future lobs after each checkpoint.
In normal application use, the DELETE statement is used to delete rows. This statement deallocates the lob space for reuse after each checkpoint.
You can design your tests in a way that recreates the database, rather than reuse the old database after removing the data.
We are obtaining a very interesting graph in our Redis architecture.
Green: master
Blue: slave
Looks like the master Redis is executing 35% more commands than the slave Redis.
It is not always the same distance.
Here is part of the log of the active redis server:
[26911] 14 Feb 13:28:44 - DB 0: 2399 keys (417 volatile) in 16384 slots HT.
[26911] 14 Feb 13:28:44 - DB 1: 498 keys (498 volatile) in 1024 slots HT.
[26911] 14 Feb 13:28:44 - DB 2: 1 keys (0 volatile) in 4 slots HT.
[26911] 14 Feb 13:28:44 - 706 clients connected (1 slaves), 33794240 bytes in use
and during the same time on the salve:
[17748] 14 Feb 13:28:44 - DB 0: 2398 keys (417 volatile) in 16384 slots HT.
[17748] 14 Feb 13:28:44 - DB 1: 497 keys (497 volatile) in 1024 slots HT.
[17748] 14 Feb 13:28:44 - DB 2: 1 keys (0 volatile) in 4 slots HT.
[17748] 14 Feb 13:28:44 - 1 clients connected (0 slaves), 24839792 bytes in use
So they look like they are almost 1:1 synchronized.
We wonder which can be the cause of this gap. Also we are asking our selves if this means there are unnecessary commands sent to Redis that we can optimize.
Here's a possible explanation: total_commands_processed reports all commands, reads, writes and server related commands. Only write commands will be propagated to the slave(s).
In a setup where you only write to the master, and read from the slave(s), you will have a higher total_commands_processed on the slave(s) (all reads + all writes).
If you write to and read from the master, and only keep the slave as a backup, or to persist to disk, the master will have a higher total_commands_processed.
In fact, it's very improbable that the master and slave will have the same number of total_commands_processed.