I'm trying to integrate Nifi with Redis in sentinel mode, as described in this tutorial:
https://bryanbende.com/development/2017/10/09/apache-nifi-redis-integration
My Redis cluster has 2 nodes, running on port 6391, and 2 sentinel, running on port 6392. It seems to work ok:
127.0.0.1:6392> sentinel master mymaster
1) "name"
2) "mymaster"
3) "ip"
4) "192.168.50.5"
5) "port"
6) "6391"
7) "runid"
8) "d8adfb30d836ad305b96d887dfe2beb74c435305"
9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "626"
19) "last-ping-reply"
20) "626"
21) "down-after-milliseconds"
22) "5000"
23) "info-refresh"
24) "5796"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "56761859"
29) "config-epoch"
30) "0"
31) "num-slaves"
32) "1"
33) "num-other-sentinels"
34) "1"
Nifi dataflow:
Redis Connection Pool:
When I run the dataflow, I get this error:
PutDistributedMapCache[id=08e39b65-0176-1000-0000-0000185bd23e] failed
to process session due to All sentinels down, cannot determine where
is mymaster master is running...; Processor Administratively Yielded
for 1 sec: redis.clients.jedis.exceptions.JedisConnectionException:
All sentinels down, cannot determine where is mymaster master is
running...
After countless hours of suffering I finally found a workaround by disabling the password in sentinel. The password to specify is that of redis, not sentinel
Related
The slowlog command on Azure redis returns the following item in response. What does this command do? It doesn't seem to be a command triggered from the client.
1) (integer) 260
2) (integer) 1660587982
3) (integer) 15508
4) 1) "ManagementCommand"
2) "list"
5) "[::]:31729"
6) ""
ManagementCommand are commands that are triggered by data plane components owned by Azure Cache for Redis for health monitoring purposes there are some Redis commands that return info about the cache itself. The slowlog is an example of this.
I have a strange issue with logrotate on a Raspbian 9 system.
Logrotate appears to be configured to rotate /var/log/syslog every seven days. When I run logrotate -f -d /etc/logrotate.conf the output tells me:
rotating pattern: /var/log/syslog
forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/syslog
Now: 2021-03-16 09:56
Last rotated at 2020-11-02 12:26
log needs rotating
rotating log /var/log/syslog, log->rotateCount is 7
dateext suffix '-20210316'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/syslog.7.gz to /var/log/syslog.8.gz (rotatecount 7, logstart 1, i 7),
renaming /var/log/syslog.6.gz to /var/log/syslog.7.gz (rotatecount 7, logstart 1, i 6),
renaming /var/log/syslog.5.gz to /var/log/syslog.6.gz (rotatecount 7, logstart 1, i 5),
renaming /var/log/syslog.4.gz to /var/log/syslog.5.gz (rotatecount 7, logstart 1, i 4),
renaming /var/log/syslog.3.gz to /var/log/syslog.4.gz (rotatecount 7, logstart 1, i 3),
renaming /var/log/syslog.2.gz to /var/log/syslog.3.gz (rotatecount 7, logstart 1, i 2),
renaming /var/log/syslog.1.gz to /var/log/syslog.2.gz (rotatecount 7, logstart 1, i 1),
renaming /var/log/syslog.0.gz to /var/log/syslog.1.gz (rotatecount 7, logstart 1, i 0),
log /var/log/syslog.8.gz doesn't exist -- won't try to dispose of it
renaming /var/log/syslog to /var/log/syslog.1
creating new /var/log/syslog mode = 0640 uid = 0 gid = 4
running postrotate script
running script with arg /var/log/syslog: "
invoke-rc.d rsyslog rotate > /dev/null
"
So it says it is renaming /var/log/syslog to /var/log/syslog.1 and creating a new syslog. So everything appears to be ok so far.
Just, it does noting. There is no syslog.1 afterwards and the syslog file is the same as before. Nothing happened.
One thing to mention: /var/log is tmpfs- is this related?
Mounted as: tmpfs on /var/log type tmpfs (rw,nosuid,nodev,relatime)
Thanks for ideas!
/KNEBB
And to update the issue here: I was so dumb not reading. When running in debug mode (as done above) logrotate does nothing- it just tells you what it would be if run without "-d".
WE have a redis cluster with 4 master and 4 slave. The 4 master are on one physical host and the slaves are on other physical host. WE have observed a frequest auto failvoer happening between the master and slave even though the servers are up and running(suspecting the network glitch here). Once they are failed over the new master CPU is going high and is throwing the redis Server exception (as attached screenshot) disconnecting the clients. Below are the cluster node details:
6adb459bc1cda0ae002109140d04015c531c6910 10.10.52.38:6379 slave 0060ee610b3a52bf88a0202aff0ce63039354578 0 1555648709383 58 connected
46f38129c861ff775badc67cc869493ee28fd166 10.10.52.44:6379 slave 19538764e5cde1014f1fd35afbf1af3a217de7b4 0 1555648708378 66 connected
1833427e42afa74273aa33696ed7e5f80f40e244 10.10.52.40:6379 master - 0 1555648706367 63 connected 6827-9556 15018-16383
a0f14e54f18e1a04c448cd09e851459863e929b0 10.10.52.42:6379 slave d190c341144350bf9dbad67841104ed75ccbdcdc 0 1555648710388 65 connected
0060ee610b3a52bf88a0202aff0ce63039354578 10.10.52.37:6379 master - 0 1555648704357 58 connected 2730-5460 13653-15017
b702bbcddb6a39e2deb8567804fa5d4468fbe5cc 10.10.52.39:6379 slave 1833427e42afa74273aa33696ed7e5f80f40e244 0 1555648708879 63 connected
d190c341144350bf9dbad67841104ed75ccbdcdc 10.10.52.41:6379 master - 0 1555648710385 65 connected 9557-13652
19538764e5cde1014f1fd35afbf1af3a217de7b4 10.10.52.43:6379 myself,master - 0 0 66 connected 0-2729 5461-6826
Also the Cluster config details:
1) "cluster-node-timeout"
2) "15000"
3) "cluster-migration-barrier"
4) "1"
5) "cluster-slave-validity-factor"
6) "10"
7) "cluster-require-full-coverage"
8) "yes"
Attached is the log from one of the slave: https://pastebin.com/GS8ChyeH
Is the configuration correct? How do we prevent this from happening?
I have the following REDIS SET:
127.0.0.1:6379[2]> hgetall available
1) "00001"
2) "unassigned"
3) "00002"
4) "unassigned"
5) "00003"
6) "unassigned"
7) "00004"
8) "unassigned"
9) "00005"
10) "unassigned"
127.0.0.1:6379[2]>
I have the following python code that uses a pipeline to "move" accounts from the available list to a reserved list / set:
def reserve_mailboxes(lower, upper):
try:
assert(lower is not None)
assert(upper is not None)
if DEBUG == True: print("reserve_mailboxes invoked with lower limit: " + str(lower) + ", upper limit: " + str(upper))
for number in range(int(lower), int(upper) + 1):
number = '{0:05d}'.format(number) #zero pad
r = redis.Redis(connection_pool=POOL)
p = r.pipeline()
p.hmset('reserved', {number:'reserved'})
p.hdel('available', {number})
response = p.execute()
logging.info(response)
if not response[1] == True:
if DEBUG == True: logging.info(response)
return True
except Exception as e:
logging.error(e)
return False
If you take a look at my code where I create and execute the pipeline, the hdel command actually has a syntax error. it really should be:
p.hdel('available', number)
However, when I run this code it goes ahead and adds 2 entries to "reserved" list ... but doesn't remove them from the available list. This is what my data looks like at this point:
127.0.0.1:6379[2]> hgetall reserved
1) "00003"
2) "reserved"
3) "00004"
4) "reserved"
127.0.0.1:6379[2]> hgetall available
1) "00001"
2) "unassigned"
3) "00002"
4) "unassigned"
5) "00003"
6) "unassigned"
7) "00004"
8) "unassigned"
9) "00005"
10) "unassigned"
The log shows the following "response" / result from the two commands:
root - INFO - [True, 0]
If the hdel worked, it should return a 1 not 0.
btw. When I remove the typo the code properly removes from one list and adds to the other.
Questions
I thought the pipeline was supposed to revert all commands if one of them fails. So in other words, since the hdel is returning 0, should it undo the hmset?
Is there a more efficient way to do this? Move a record from one set to another?
Regardless what you thought, a pipeline is just a way to batch operations w/o waiting for the replies. The failure of one or more operations in the pipeline does not prevent either rolls back or other operations in it.
These aren't Sets, but Hashes. You can look into Lua scripting (see the EVAL command) to optimize the performance of your logic by running it within the Redis server.
You can wrap the pipelined commands with MULTI and EXEC to make them "atomic".
For example:
MULTI
INCR pipeline_counter
EXPIRE pipeline_counts 3600
EXEC
See the doc of EXEC command for more information.
Problem
I have a situation where data that I created on the master doesn't seem to have been replicated to my slaves properly.
Master Redis DB Setup Info
I have a master running on 10.1.1.1. The configuration is set to "SAVE" to disk. Here's a snippet from the configuration file:
save 900 1
save 300 10
save 60 10000
When I run a scan command against the hash in question, here are the results (which are correct):
127.0.0.1:6379> scan 0 match dep:*
1) "13"
2) 1) "dep:+19999999999_00:00_00:00"
2) "dep:+19999999999_19:00_25:00"
3) "dep:+19999999999_08:00_12:00"
127.0.0.1:6379>
Slave 1 Setup
Slave 1 has been set up to run in memory only. So in the configuration file, all the "save" options have been commented out.
Here's the data I have in slave 1: (missing a record)
127.0.0.1:6379> scan 0 match dep:*
1) "15"
2) 1) "dep:+19999999999_00:00_00:00"
2) "dep:+19999999999_19:00_25:00"
127.0.0.1:6379>
When I run the "info" command on this slave, this is what I get back: (only picked specific items that I thought might pertain to this problem)
# Replication
role:slave
master_host:10.1.1.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:5
master_sync_in_progress:0
slave_repl_offset:346292
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
#Stats
expired_keys:0
#Persistence
aof_enabled:0
Slave 2 Setup
Slave 2 is also supposed to be an in memory data store only. So all the save options in the config file have also been commented out like so:
#save 900 1
#save 300 10
#save 60 10000
This is the data I have on slave 2 (notice that it's missing data, but different records from slave 1)
127.0.0.1:6379> scan 0 match dep:*
1) "3"
2) 1) "dep:+19999999999_00:00_00:00"
2) "dep:+19999999999_08:00_12:00"
127.0.0.1:6379>
Some of the results from the info command:
# Replication
role:slave
master_host:10.1.1.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_repl_offset:346754
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
#Stats
expired_keys:0
#Persistence
aof_enabled:0
This is my first crack at using REDIS, so I'm sure it's something simple that I've missed.
I haven't tried to restart REDIS on the slaves just yet, because I don't want to lose any artifacts that might help me troubleshoot / understand how I got myself here in the first place.
Any suggestions would be appreciated.
EDIT 1
In checking the logs on slave 2, this is what I found:
4651:S 27 Sep 18:39:27.197 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
4651:S 27 Sep 18:39:27.197 # Server started, Redis version 3.0.5
4651:S 27 Sep 18:39:27.197 * The server is now ready to accept connections on port 6379
4651:S 27 Sep 18:39:27.198 * Connecting to MASTER 10.1.1.1:6379
4651:S 27 Sep 18:39:27.198 * MASTER <-> SLAVE sync started
4651:S 27 Sep 18:40:28.284 # Timeout connecting to the MASTER...
4651:S 27 Sep 18:40:28.284 * Connecting to MASTER 10.1.1.1:6379
4651:S 27 Sep 18:40:28.284 * MASTER <-> SLAVE sync started
4651:S 27 Sep 18:41:29.369 # Timeout connecting to the MASTER...
4651:S 27 Sep 18:41:29.369 * Connecting to MASTER 10.1.1.1:6379
4651:S 27 Sep 18:41:29.369 * MASTER <-> SLAVE sync started
4651:S 27 Sep 18:42:00.452 * Non blocking connect for SYNC fired the event.
4651:S 27 Sep 18:42:00.453 * Master replied to PING, replication can continue...
4651:S 27 Sep 18:42:00.453 * Partial resynchronization not possible (no cached master)
4651:S 27 Sep 18:42:00.463 * Full resync from master: b46c3622e4ef4c5586ebd2ec23eabcb04c3fcf32:1
4651:S 27 Sep 18:42:00.592 * MASTER <-> SLAVE sync: receiving 173 bytes from master
4651:S 27 Sep 18:42:00.592 * MASTER <-> SLAVE sync: Flushing old data
4651:S 27 Sep 18:42:00.592 * MASTER <-> SLAVE sync: Loading DB in memory
4651:S 27 Sep 18:42:00.592 * MASTER <-> SLAVE sync: Finished with success
How do redis slaves recover when there is a time out connecting to the master? I'm also wondering what this error means "Partial resynchronization not possible (no cached master)".
Currently googling... But if you have any comments, please feel free
EDIT 2
Here's another really interesting find (at least for me).
I just added a new item the master, like so:
127.0.0.1:6379> HMSET dep:+19999999999_15:00_18:45:00 ext 2222 dd me.net days "fri"
OK
127.0.0.1:6379> scan 0 match dep:*
1) "13"
2) 1) "dep:+19999999999_00:00_00:00"
2) "dep:+19999999999_19:00_25:00"
3) "dep:+19999999999_15:00_18:45:00"
4) "dep:+19999999999_08:00_12:00"
127.0.0.1:6379>
And now, when i check slave one again, it still only has 2 records, but its dropped a record that used to be there, and replaced it with the new i just added:
127.0.0.1:6379> scan 0 match dep:*
1) "7"
2) 1) "dep:+19999999999_00:00_00:00"
2) "dep:+19999999999_15:00_18:45:00"
127.0.0.1:6379>
EDIT 3
From the answer below, it looks like the first number returned by the SCAN command is a position in the cursor... And in reading the documentation I can specify a count indicating the number of records to return.
But this still raises some questions for me. For example, in line with the answer below, I tried the following SCAN command on a slave:
127.0.0.1:6379> scan 0 match dep:*
1) "7"
2) 1) "dep:+19999999999_00:00_00:00"
2) "dep:+19999999999_15:00_18:45:00"
127.0.0.1:6379> scan 7 match dep:*
1) "0"
2) 1) "dep:+19999999999_19:00_25:00"
2) "dep:+19999999999_08:00_12:00"
127.0.0.1:6379>
This makes sense to me... it seems to be returning 2 records at a time ( still need to figure out how I can change this default)
According to this post - Redis scan count: How to force SCAN to return all keys matching a pattern? - , I can use the "count" keyword to indicate how many records to return.
But in order to get all 4 of the records I have, I had to run several queries before the cursor value came back as a zero... and i dont' know why. For example:
127.0.0.1:6379> scan 0 match dep:* count 3
1) "10"
2) 1) "dep:+19999999999_00:00_00:00"
127.0.0.1:6379> scan 10 match dep:* count 3
1) "3"
2) (empty list or set)
127.0.0.1:6379> scan 3 match dep:* count 3
1) "7"
2) 1) "dep:+19999999999_15:00_18:45:00"
127.0.0.1:6379> scan 7 match dep:* count 3
1) "0"
2) 1) "dep:+19999999999_19:00_25:00"
2) "dep:+19999999999_08:00_12:00"
127.0.0.1:6379>
Why didn't the first request return 3 records? In my mind, at most, I should have had to run this scan command 2 times.
Can you explain what's happening here?
Also, maybe I shouldn't be using the scan command in my node js REST API? Imagine that a user will make a request for widget information... and I need to query this hash to find the key. It feels like this type of iteration would be very inefficient. The KEYS command will work too, but as per the docs,I shouldn't be using that in production because it will affect performance.
Any comments / insights would be appreciated.
You haven't iterate all keys from Redis instance.
In order to do a full iteration, you should continue sending the SCAN command to Redis with the returned cursor, until the returned cursor is 0.
In your last example:
127.0.0.1:6379> scan 0 match dep:*
1) "7" <---- returned cursor
2) 1) "dep:+19999999999_00:00_00:00"
2) "dep:+19999999999_15:00_18:45:00"
127.0.0.1:6379>
// here, you need continue sending scan command with the returned cursor, i.e. 7
127.0.0.1:6379> scan 7 match dep:*
// then you can get more results from Redis
// If the full iteration is finished, it should return something like this:
1) "0" <----- this means the full iteration is finished
2) 1) "dep:more result"
2) "dep:last result"
Edit
The count number for SCAN command is just a hint. There's no guarantee that Redis should return exactly count number results (see the doc for more details).
In order to get all keys in one shot, you can use the KEYS command. However, just as you mentioned, it's NOT a good idea (it might block Redis for a long time), and that's why Redis has this SCAN command to get all keys in iteration.
Both SCAN and KEYS commands traverse the whole key space to find the match. So if the data set is very large, they both take a long time to get/iterate all keys.
From your problem description, I think you should store your data in Redis' HASH structure, and use HKEYS, HGETALL or HSCAN to get the data:
hset dep:+19999999999 00:00_00:00:00 value
hset dep:+19999999999 15:00_18:45:00 value
hkeys dep:+19999999999 <----- get all fields in this hash
hgetall dep:+19999999999 <----- get all fields and values in this hash
hscan dep:+19999999999 0 <----- scan the hash to key fields
This should be much more efficient than traverse the whole key space. Especially, if there's not too much fields in the hash, HKEYS and HGETALL can get all keys/data in one shot, and very fast. However, if there's too much fields in the hash, you still need to use HSCAN to do iterations.