When I run systemctl status on redis service, why do I at times see status as "Finished with success" and at times "Partial Resynchronization accepted". Why/when do we get partial resynchronization?
Status: "MASTER <-> REPLICA sync: Finished with success. Ready to accept connections in read-write mode."
vs
Status: "MASTER <-> REPLICA sync: Partial Resynchronization accepted. Ready to accept connections in read-write mode."
Tried bouncing redis process and it started showing "Finished with success" once but not other times.
Related
<redisson.version>3.16.3</redisson.version>
readMode: SLAVE
Not sure why am I seeing this error.
caused by: org.redisson.client.RedisException: ERR WAIT cannot be used with slave instances. Please also note that since Redis 4.0 if a slave is configured to be writable (which is not the default) writes to slaves are just local and are not propagated.. channel: [id: 0x90c58500, L:/172.16.189.108:43192 - R:10.112.94.132/10.112.94.132:15003] command: (WAIT), promise: RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise#35eeb35e(incomplete)], params: [1, 1000]
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:370)
at org.redisson.client.handler.CommandDecoder.decodeCommandBatch(CommandDecoder.java:271)
at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:210)
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:137)
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:113)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
How come celery cannot find my sentinel service?
I have:
app.conf.broker_url = "sentinel://192.168.29.11:26379"
app.conf.broker_transport_options = {"master_name": "mymaster"}
And what I am getting is:
{"message": "consumer: Cannot connect to sentinel://192.168.29.11:26379: No master found for None.
Trying again in 2.00 seconds... (1/100)",
"level": "ERROR",
"logger": "celery.worker.consumer.consumer"}
Why is there "No master found for None" when I am specifying the master_name?
It was my mistake, had another app.conf.broker_transport_options defined elsewhere and it overwrote the setting.
I have a Redis v3.0.2 Master and Slave
on the slave I issue slaveof (masterIP) 6379
Sync happens, logs looking fine on master and slave. Key counts look sane.
After the sync completes and the slave loads the database, No more operations happen.
running monitor on the master give me hundreds of sets / sec.
the slave only sees a few deletes and an occasional PING
Slave Log:
2734:S 16 Aug 07:23:29.460 * MASTER <-> SLAVE sync: Loading DB in memory
2734:S 16 Aug 07:25:16.531 * MASTER <-> SLAVE sync: Finished with success
Slave Monitor:
~
[119](root#[slave])[0]:\> redis-cli
127.0.0.1:6379> monitor
OK
1534405063.907020 [0 [master]:6379] "PING"
1534405065.409863 [0 [master]:6379] "DEL" "pmlock12"
1534405065.709784 [0 [master]:6379] "DEL" "pmlock22"
1534405065.909400 [0 [master]:6379] "DEL" "pmlock27"
Master Log
2951:C 16 Aug 07:20:57.908 * RDB: 279 MB of memory used by copy-on-write
2745:M 16 Aug 07:20:58.297 * Background saving terminated with success
2745:M 16 Aug 07:22:59.369 * Synchronization with slave 10.168.230.15:6379 succeeded
Master Monitor:
1534405287.136316 [0 [src]:54660] "SET" "CMP36" "{\"m_cur...
1534405252.002731 [0 [src]:45742] "SET" "PM14" "H4sIAAAAAAAAAO1cW4...
Master Info
[209](root#master)[0]:\> redis-cli info replication
# Replication
role:master
connected_slaves:1
slave0:ip=[slave],port=6379,state=online,offset=1747897005,lag=0
master_repl_offset:1748304094
repl_backlog_active:1
repl_backlog_size:104857600
repl_backlog_first_byte_offset:1643446495
repl_backlog_histlen:104857600
I've rebooted the master and slave, i just can't get the master to send through anything but ping and delete. I'm not well versed on redis so i'm sure i'm just missing something.
Slave status :
Last_IO_Errno: 1595
Last_IO_Error: Relay log write failure: could not queue event from master
Last_SQL_Errno: 0
from error log :
[ERROR] Slave I/O for channel 'db12': Unexpected master's heartbeat data: heartbeat is not compatible with local info; the event's data: log_file_name toku10-bin.000063<D1> log_pos 97223067, Error_code: 1623
[ERROR] Slave I/O for channel 'db12': Relay log write failure: could not queue event from master, Error_code: 1595
I tried to restarting the slave_io thread for many times, still its same.
we need to keep on start io_thread whenever it stopped manually, hope its bug from percona
I have simply written shell and scheduled the same for every 10mins to check if io_thread is not running , start slave io_thread for channel 'db12';. It's working as of now
I just installed OpenStack Juno using devstack, and observed that RabbitMQ (package rabbitmq-server-3.1.5-10 installed by yum) is not stable, i.e. it quickly eats up the memory and shuts down; there is 2G of RAM. Below is the messages from logs and 'systemctl status' before the daemon died:
=INFO REPORT==== 18-Dec-2014::01:25:40 ===
vm_memory_high_watermark clear. Memory used:835116352 allowed:835212083
=WARNING REPORT==== 18-Dec-2014::01:25:40 ===
memory resource limit alarm cleared on node rabbit#node
=INFO REPORT==== 18-Dec-2014::01:25:40 ===
accepting AMQP connection <0.27011.5> (10.0.0.11:55198 -> 10.0.0.11:5672)
=INFO REPORT==== 18-Dec-2014::01:25:41 ===
vm_memory_high_watermark set. Memory used:850213192 allowed:835212083
=WARNING REPORT==== 18-Dec-2014::01:25:41 ===
memory resource limit alarm set on node rabbit#node.
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
rabbitmqctl[770]: ===========
rabbitmqctl[770]: nodes in question: [rabbit#node]
rabbitmqctl[770]: hosts, their running nodes and ports:
rabbitmqctl[770]: - node: [{rabbitmqctl770,40089}]
rabbitmqctl[770]: current node details:
rabbitmqctl[770]: - node name: rabbitmqctl770#node
rabbitmqctl[770]: - home dir: /var/lib/rabbitmq
rabbitmqctl[770]: - cookie hash: FftrRFUESg4RKWsyb1cPqw==
systemd[1]: rabbitmq-server.service: control process exited, code=exited status=2
systemd[1]: Unit rabbitmq-server.service entered failed state.
I know about set_vm_memory_high_watermark, but it doesn't solve the issue. I want to ensure that the daemon doesn't shut down abruptly. I wonder if someone saw this before and could advise?
Thanks.
UPDATE
Upgraded to version 3.4.2 taken directly from www.rabbitmq.com/download.html The new version doesn't consume RAM that fast and tends to work longer then previous version, but eventually still eats out all the memory and shuts.
I think the number of connections in the servers are increasing and they are being held like that without closing that's why it is consuming more memory. When the usage of RAM increases beyond the watermark rabbitmq server won't accept any network request. Either you have to close the connections which all are opened or you have to increase the RAM of the system. But increasing the RAM will only reduce the problem for some time but you'll face the problem again it is better to close the connections.
try to use CloudAMQP instead of installing locally. This will be fixed then. firstly create a rabbitMQ account here. https://customer.cloudamqp.com/signup.
then create your queue there and connect with your application.