I am facing a peculiar issue.
In my redis.conf file, at first I enable a password authentication by setting:
requirepass admin
When I connect to Redis via CLI, I am required to authenticate myself before continuing any operation, so all good so far:
127.0.0.1:6379> get name
(error) NOAUTH Authentication required.
127.0.0.1:6379> auth admin
OK
127.0.0.1:6379> get name
"sahay"
Now, I try to create a ACL user by running:
127.0.0.1:6379> acl setuser nonadminuser on >generalpassword +#all -#dangerous ~*
OK
So far so good, now I want to persist this new user to a ACL file, so I run:
127.0.0.1:6379> acl save
(error) ERR This Redis instance is not configured to use an ACL file. You may want to specify users via the ACL SETUSER command and then issue a CONFIG REWRITE (assuming you have a Redis configuration file set) in order to store users in the Redis configuration.
This is also OK, since I have not set any configuration of aclfile in my redis.conf.
So, I stop my redis server and add this line in my redis.conf file:
aclfile /Ankit/redis_installation/redis-stable/acl_users.acl
I also create a acl_users.acl file in the above mentioned directory, because without it Redis throws an error that no such file exists.
Now comes the peculiar part. When I start redis, and connect to it via CLI, it doesn't ask me to authenticate! Even though
requirepass admin
is set in the redis.conf file. In fact, it throws error when I try to run a password.
127.0.0.1:6379> get name
"sahay"
127.0.0.1:6379> auth admin
(error) ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?
Also, when I do an ACL save now (after creating the new user), that new user is created and persisted in acl file but the strange thing is default user is stored with "nopass"
Why is default user configured as nopass even when config file has a requirepass clause?
requirepass is not compatible with ACL feature. If you config with ACL rules, requirepass is ignored. That's why you can operate Redis as the default user without sending auth command.
With ACL feature, you need to explicitly set a rule for default user.
user default on +#all ~* >password
I am trying to limit the allowed privileges for external redis sentinel users by renaming critical commands as follow:
sentinel rename-command mymaster FAILOVER failover-secret
However, the configurations are being ignored, and I still can trigger the renamed command using the original name:
127.0.0.1:26379> sentinel failover mymaster
OK
Redis Version:
Redis server v=6.0.9 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=e874f7259751a389
The best option would be to put this in your Redis server's config file as opposed to setting it via CLI. It sounds like setting it this way either only applies to that connection (so other connections won't have that config change) or it only persists until the server restarts. Putting it in the config file would persist for all connections, and across restarts.
Another option if you're using Redis v6 (or can upgrade to v6) is to create separate users and specify the available commands per user. This option is discussed in this answer.
I am looking at a project to persist data realtime and am considering Redis.
Issue I see with it is that anyone can issue a command to drop all the data at once
How not to allow user to issue such a command before everything is persisted successfully.
You can use rename-command in your config. According to the security section of redis.conf:
# Command renaming.
#
# It is possible to change the name of dangerous commands in a shared
# environment. For instance the CONFIG command may be renamed into something
# hard to guess so that it will still be available for internal-use tools
# but not available for general clients.
#
# Example:
#
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
#
# It is also possible to completely kill a command by renaming it into
# an empty string:
#
# rename-command CONFIG ""
#
# Please note that changing the name of commands that are logged into the
# AOF file or transmitted to replicas may cause problems.
On top of renaming the command, as explained in #Leisen Chang's answer, in the soon-to-be-released version 6 of Redis, ACL (shorts for Access Control List) will be introduced.
ACL allows you to create users and assign them with permissions for calling commands and/or accessing key patterns. For example, here's how you'd create a user that can do everything except FLUSHALL:
redis> ACL SETUSER myuser on >mypass +#all -flushall
Setup: I have three machines with co-located Sentinel/Redis, exactly as in example-2, configured with sentinel auth-pass, masterauth, requirepass. -- failover, read/write -- all works.
Requirement: I want to add in a Redis slave (4th instance of Redis) that does not require auth, but replicates from the main cluster.
Issue: I setup exactly as in the uncommon case where you need a slave that is accessible without authentication, ie, Master >> Slave,replica-priority 0.
When using redis-py client, I ask Sentinel for a slave to read from and this no-auth-slave is chosen, i receive error: Client sent AUTH, but no password is set and connection is aborted. I believe the client is just passing back the error from server.
I've Tried:
Master >> Slave,replica-priority 0 >> Slave,replica-priority 0. All clients, servers, sync work (because Sentinel doesn't know about this readonly slave) except I get +fix-slave-config entries every 10s in Sentinel log. Not sure if this is concerning??
Setup as defined in "the uncommon case..." with Master >> (Slave,replica-priority 0) but client error, and unable to proceed with connection/request.
Questions:
Is Master >> Slave >> Slave with Sentinel +fix-slave-config entries ok?
Is Client sent AUTH, but no password is set a bug/feature?
I would definitely prefer to have all Redis slaves known by Sentinel for HA though, but doesn't work mixing auth/no-auth.
Redis 5.0.3, redis-py 3.2.0
I am running a spring boot service using spring data redis and here is the following configuration.
The service seems to work but I am seeing a stream of Lost Sentinel messages in the logs. The sentinel nodes are reachable form the VM where I am running the service. I was able to telnet to them directly from that VM. Any idea why this is happening?
spring:
profiles:
active: core-perf,swagger
default: core-perf,swagger
redis:
Pool: #Pool properties
# Max number of "idle" connections in the pool. Use a negative value to indicate
# an unlimited number of idle connections.
maxIdle: 8
# Target for the minimum number of idle connections to maintain in the pool.
# This setting only has an effect if it is positive.
minIdle: 0
# Max number of connections that can be allocated by the pool at a given time. Use a negative value for no limit.
maxActive: 8
# Maximum amount of time (in milliseconds) a connection allocation should block
# before throwing an exception when the pool is exhausted. Use a negative value
# to block indefinitely.
maxWait: -1
sentinel: #Redis sentinel properties.
master: mymaster
nodes: 10.202.56.209:26379, 10.202.56.213:26379, 10.202.58.80:26379
2015-06-15 17:30:54.896 ERROR 6677 --- [Thread-9] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.58.80:26379. Sleeping 5000ms and retrying.
2015-06-15 17:30:59.894 ERROR 6677 --- [Thread-8] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.56.213:26379. Sleeping 5000ms and retrying.
2015-06-15 17:30:59.897 ERROR 6677 --- [Thread-9] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.58.80:26379. Sleeping 5000ms and retrying.
2015-06-15 17:31:04.975 ERROR 6677 --- [Thread-9] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.58.80:26379. Sleeping 5000ms and retrying.
2015-06-15 17:31:04.976 ERROR 6677 --- [Thread-8] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.56.213:26379. Sleeping 5000ms and retrying.
2015-06-15 17:31:09.976 ERROR 6677 --- [Thread-9] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.58.80:26379. Sleeping 5000ms and retrying.
2015-06-15 17:31:09.976 ERROR 6677 --- [Thread-8] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.56.213:26379. Sleeping 5000ms and retrying.
2015-06-15 17:31:15.054 ERROR 6677 --- [Thread-8] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.56.213:26379. Sleeping 5000ms and retrying.
2015-06-15 17:31:15.055 ERROR 6677 --- [Thread-9] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.58.80:26379. Sleeping 5000ms and retrying.
2015-06-15 17:31:20.055 ERROR 6677 --- [Thread-8] redis.clients.jedis.JedisSentinelPool : Lost connection to Sentinel at 10.202.56.213:26379. Sleeping 5000ms and retrying.
We discovered the issue. There was a blank between the node pairs in the application.yml and once we removed this " " the Lost Sentinel log message disappeared.
so from
nodes: 10.202.56.209:26379, 10.202.56.213:26379, 10.202.58.80:26379
to
nodes: 10.202.56.209:26379,10.202.56.213:26379,10.202.58.80:26379
It would probably be a good thing if the committers looked at this as it would seem to be somewhat mysterious for users.
I lost my head over this issue for 2 days until I decided to reinstall it again and then configure it via my old config files and then after replacing sentinel.conf file with the below text.
It finally worked.
# *** IMPORTANT ***
#
# By default Sentinel will not be reachable from interfaces different than
# localhost, either use the 'bind' directive to bind to a list of network
# interfaces, or disable protected mode with "protected-mode no" by
# adding it to this configuration file.
#
# Before doing that MAKE SURE the instance is protected from the outside
# world via firewalling or other means.
#
# For example you may use one of the following:
#
# bind 127.0.0.1 192.168.1.1
#
# protected-mode no
# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379
# By default Redis Sentinel does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/redis-sentinel.pid when
# daemonized.
daemonize no
# When running daemonized, Redis Sentinel writes a pid file in
# /var/run/redis-sentinel.pid by default. You can specify a custom pid file
# location here.
pidfile /var/run/redis-sentinel.pid
# Specify the log file name. Also the empty string can be used to force
# Sentinel to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile ""
# sentinel announce-ip <ip>
# sentinel announce-port <port>
#
# The above two configuration directives are useful in environments where,
# because of NAT, Sentinel is reachable from outside via a non-local address.
#
# When announce-ip is provided, the Sentinel will claim the specified IP address
# in HELLO messages used to gossip its presence, instead of auto-detecting the
# local address as it usually does.
#
# Similarly when announce-port is provided and is valid and non-zero, Sentinel
# will announce the specified TCP port.
#
# The two options don't need to be used together, if only announce-ip is
# provided, the Sentinel will announce the specified IP and the server port
# as specified by the "port" option. If only announce-port is provided, the
# Sentinel will announce the auto-detected local IP and the specified port.
#
# Example:
#
# sentinel announce-ip 1.2.3.4
# dir <working-directory>
# Every long running process should have a well-defined working directory.
# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
# for the process to don't interfere with administrative tasks such as
# unmounting filesystems.
dir /tmp
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
#
# Tells Sentinel to monitor this master, and to consider it in O_DOWN
# (Objectively Down) state only if at least <quorum> sentinels agree.
#
# Note that whatever is the ODOWN quorum, a Sentinel will require to
# be elected by the majority of the known Sentinels in order to
# start a failover, so no failover can be performed in minority.
#
# Replicas are auto-discovered, so you don't need to specify replicas in
# any way. Sentinel itself will rewrite this configuration file adding
# the replicas using additional configuration options.
# Also note that the configuration file is rewritten when a
# replica is promoted to master.
#
# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
sentinel monitor mymaster 127.0.0.1 6379 2
# sentinel auth-pass <master-name> <password>
#
# Set the password to use to authenticate with the master and replicas.
# Useful if there is a password set in the Redis instances to monitor.
#
# Note that the master password is also used for replicas, so it is not
# possible to set a different password in masters and replicas instances
# if you want to be able to monitor these instances with Sentinel.
#
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
#
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
# sentinel auth-user <master-name> <username>
#
# This is useful in order to authenticate to instances having ACL capabilities,
# that is, running Redis 6.0 or greater. When just auth-pass is provided the
# Sentinel instance will authenticate to Redis using the old "AUTH <pass>"
# method. When also an username is provided, it will use "AUTH <user> <pass>".
# In the Redis servers side, the ACL to provide just minimal access to
# Sentinel instances, should be configured along the following lines:
#
# user sentinel-user >somepassword +client +subscribe +publish \
# +ping +info +multi +slaveof +config +client +exec on
# sentinel down-after-milliseconds <master-name> <milliseconds>
#
# Number of milliseconds the master (or any attached replica or sentinel) should
# be unreachable (as in, not acceptable reply to PING, continuously, for the
# specified period) in order to consider it in S_DOWN state (Subjectively
# Down).
#
# Default is 30 seconds.
sentinel down-after-milliseconds mymaster 30000
# IMPORTANT NOTE: starting with Redis 6.2 ACL capability is supported for
# Sentinel mode, please refer to the Redis website https://redis.io/topics/acl
# for more details.
# Sentinel's ACL users are defined in the following format:
#
# user <username> ... acl rules ...
#
# For example:
#
# user worker +#admin +#connection ~* on >ffa9203c493aa99
#
# For more information about ACL configuration please refer to the Redis
# website at https://redis.io/topics/acl and redis server configuration
# template redis.conf.
# ACL LOG
#
# The ACL Log tracks failed commands and authentication events associated
# with ACLs. The ACL Log is useful to troubleshoot failed commands blocked
# by ACLs. The ACL Log is stored in memory. You can reclaim memory with
# ACL LOG RESET. Define the maximum entry length of the ACL Log below.
acllog-max-len 128
# Using an external ACL file
#
# Instead of configuring users here in this file, it is possible to use
# a stand-alone file just listing users. The two methods cannot be mixed:
# if you configure users here and at the same time you activate the external
# ACL file, the server will refuse to start.
#
# The format of the external ACL user file is exactly the same as the
# format that is used inside redis.conf to describe users.
#
# aclfile /etc/redis/sentinel-users.acl
# requirepass <password>
#
# You can configure Sentinel itself to require a password, however when doing
# so Sentinel will try to authenticate with the same password to all the
# other Sentinels. So you need to configure all your Sentinels in a given
# group with the same "requirepass" password. Check the following documentation
# for more info: https://redis.io/topics/sentinel
#
# IMPORTANT NOTE: starting with Redis 6.2 "requirepass" is a compatibility
# layer on top of the ACL system. The option effect will be just setting
# the password for the default user. Clients will still authenticate using
# AUTH <password> as usually, or more explicitly with AUTH default <password>
# if they follow the new protocol: both will work.
#
# New config files are advised to use separate authentication control for
# incoming connections (via ACL), and for outgoing connections (via
# sentinel-user and sentinel-pass)
#
# The requirepass is not compatable with aclfile option and the ACL LOAD
# command, these will cause requirepass to be ignored.
# sentinel sentinel-user <username>
#
# You can configure Sentinel to authenticate with other Sentinels with specific
# user name.
# sentinel sentinel-pass <password>
#
# The password for Sentinel to authenticate with other Sentinels. If sentinel-user
# is not configured, Sentinel will use 'default' user with sentinel-pass to authenticate.
# sentinel parallel-syncs <master-name> <numreplicas>
#
# How many replicas we can reconfigure to point to the new replica simultaneously
# during the failover. Use a low number if you use the replicas to serve query
# to avoid that all the replicas will be unreachable at about the same
# time while performing the synchronization with the master.
sentinel parallel-syncs mymaster 1
# sentinel failover-timeout <master-name> <milliseconds>
#
# Specifies the failover timeout in milliseconds. It is used in many ways:
#
# - The time needed to re-start a failover after a previous failover was
# already tried against the same master by a given Sentinel, is two
# times the failover timeout.
#
# - The time needed for a replica replicating to a wrong master according
# to a Sentinel current configuration, to be forced to replicate
# with the right master, is exactly the failover timeout (counting since
# the moment a Sentinel detected the misconfiguration).
#
# - The time needed to cancel a failover that is already in progress but
# did not produced any configuration change (SLAVEOF NO ONE yet not
# acknowledged by the promoted replica).
#
# - The maximum time a failover in progress waits for all the replicas to be
# reconfigured as replicas of the new master. However even after this time
# the replicas will be reconfigured by the Sentinels anyway, but not with
# the exact parallel-syncs progression as specified.
#
# Default is 3 minutes.
sentinel failover-timeout mymaster 180000
# SCRIPTS EXECUTION
#
# sentinel notification-script and sentinel reconfig-script are used in order
# to configure scripts that are called to notify the system administrator
# or to reconfigure clients after a failover. The scripts are executed
# with the following rules for error handling:
#
# If script exits with "1" the execution is retried later (up to a maximum
# number of times currently set to 10).
#
# If script exits with "2" (or an higher value) the script execution is
# not retried.
#
# If script terminates because it receives a signal the behavior is the same
# as exit code 1.
#
# A script has a maximum running time of 60 seconds. After this limit is
# reached the script is terminated with a SIGKILL and the execution retried.
# NOTIFICATION SCRIPT
#
# sentinel notification-script <master-name> <script-path>
#
# Call the specified notification script for any sentinel event that is
# generated in the WARNING level (for instance -sdown, -odown, and so forth).
# This script should notify the system administrator via email, SMS, or any
# other messaging system, that there is something wrong with the monitored
# Redis systems.
#
# The script is called with just two arguments: the first is the event type
# and the second the event description.
#
# The script must exist and be executable in order for sentinel to start if
# this option is provided.
#
# Example:
#
# sentinel notification-script mymaster /var/redis/notify.sh
# CLIENTS RECONFIGURATION SCRIPT
#
# sentinel client-reconfig-script <master-name> <script-path>
#
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
#
# The following arguments are passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
#
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected replica
# (now a master).
#
# This script should be resistant to multiple invocations.
#
# Example:
#
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
# SECURITY
#
# By default SENTINEL SET will not be able to change the notification-script
# and client-reconfig-script at runtime. This avoids a trivial security issue
# where clients can set the script to anything and trigger a failover in order
# to get the program executed.
sentinel deny-scripts-reconfig yes
# REDIS COMMANDS RENAMING
#
# Sometimes the Redis server has certain commands, that are needed for Sentinel
# to work correctly, renamed to unguessable strings. This is often the case
# of CONFIG and SLAVEOF in the context of providers that provide Redis as
# a service, and don't want the customers to reconfigure the instances outside
# of the administration console.
#
# In such case it is possible to tell Sentinel to use different command names
# instead of the normal ones. For example if the master "mymaster", and the
# associated replicas, have "CONFIG" all renamed to "GUESSME", I could use:
#
# SENTINEL rename-command mymaster CONFIG GUESSME
#
# After such configuration is set, every time Sentinel would use CONFIG it will
# use GUESSME instead. Note that there is no actual need to respect the command
# case, so writing "config guessme" is the same in the example above.
#
# SENTINEL SET can also be used in order to perform this configuration at runtime.
#
# In order to set a command back to its original name (undo the renaming), it
# is possible to just rename a command to itself:
#
# SENTINEL rename-command mymaster CONFIG CONFIG
# HOSTNAMES SUPPORT
#
# Normally Sentinel uses only IP addresses and requires SENTINEL MONITOR
# to specify an IP address. Also, it requires the Redis replica-announce-ip
# keyword to specify only IP addresses.
#
# You may enable hostnames support by enabling resolve-hostnames. Note
# that you must make sure your DNS is configured properly and that DNS
# resolution does not introduce very long delays.
#
SENTINEL resolve-hostnames no
# When resolve-hostnames is enabled, Sentinel still uses IP addresses
# when exposing instances to users, configuration files, etc. If you want
# to retain the hostnames when announced, enable announce-hostnames below.
#
SENTINEL announce-hostnames no