I'm trying to setup a redis cluster using redis 6.0.10 compiled with TLS support. This is part of config related to TLS and cluster :
port 0
tls-port 6390
tls-cert-file /usr/local/redis6/tls/redisTLS.pem
tls-key-file /usr/local/redis6/tls/redis.key
tls-ca-cert-file /usr/local/redis6/tls/redistlschain.crt
tls-replication yes
tls-cluster yes
tls-auth-clients no
cluster-enabled yes
also i'm using requirepass option. I have 3 nodes each has 2 instances. Each has similar config with the same certs and key.
When i try to create a cluster with following command:
/usr/local/redis6/bin/redis-cli --tls --cert /usr/local/redis6/tls/redisTLS.pem --key /usr/local/redis6/tls/redis.key --cacert /usr/local/redis6/tls/redistlschain.crt -a somepass -p 6390 --cluster create 10.1.22.11:6390 10.1.22.11:6391 10.1.22.12:6390 10.1.22.12:6391 10.1.22.13:6390 10.1.22.13:6391 --cluster-replicas 1
it stuck on connecting nodes. In the log files i see following errors:
Accepting cluster node connection from 10.1.22.11:41398
Error accepting cluster node connection: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
It seems like an issue with certificate but when i use those certificates with redis-cli utility or other redis clients they work successfully. I can connect to redis instance and execute commands.
So it's not clear what should be fixed with certificates setup.
Is it something wrong with server certificate ? Or i need a client certificate in addition?
I have an option tls-auth-clients set to "no" so expected that client certificate is not needed ( also docs mention this).
Also do i need to have different server certificate for each node or it's ok to have the same on all nodes?
Any help would be appreciated.
Thanks
Related
I have a 3-instance high availability redis deployed. On each server I have redis and sentinel installed. I am trying to set a password
so that it requests it at the moment of entering with the command "redis-cli".
I am modifying the value of the "requirepass" parameter of the "redis.conf" file.
requirepass password123
Also inside the redis terminal, I am setting the password with the following commands
config set requirepass password123
auth password123
When I connect with the following command
redis-cli --tls --cert /<path>/redis.crt --key /<path>/redis.key --cacert /<path>/ca.crt -a password123
It works fine, my problem is when I restart the redis service, for some reason the password settings are not kept and I get the following message
Warning: AUTH failed
I do not know what configuration I need to do so that the change is maintained after restarting the redis service.
The version of redis that I have installed is "Redis server v=6.0.6"
Check your ACL configuration,Your requirepass configuration will be ignored with ACL operation. I get follow infomation from redis.conf example file.
IMPORTANT NOTE: starting with Redis 6 "requirepass" is just a compatibility
layer on top of the new ACL system. The option effect will be just setting
the password for the default user. Clients will still authenticate using
AUTH as usually, or more explicitly with AUTH default
if they follow the new protocol: both will work.
The requirepass is not compatable with aclfile option and the ACL LOAD
command, these will cause requirepass to be ignored.
config rewrite
This command will solve your issue of nopass after restart.
After setting the requirepass from redis cli.
I am trying to get Redis 6 (with TLS enabled during compilation, tests after compilation were successful) to work. I am using Lets Encrypt certificate and following configuration:
tls-port 63790
tls-cert-file /etc/letsencrypt/live/myserver.net/cert.pem
tls-key-file /etc/letsencrypt/live/myserver.net/privkey.pem
tls-ca-cert-dir /etc/letsencrypt/live/myserver.net/
tls-auth-clients no
tls-protocols "TLSv1.2 TLSv1.3"
and this client command from localhost
redis-cli --tls --cert /etc/letsencrypt/live/myserver.net/cert.pem --key /etc/letsencrypt/live/myserver.net/privkey.pem --cacert /etc/letsencrypt/live/myserver.net/fullchain.pem -h myserver.net -p 63790 -a password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at myserver.net:63790: SSL_connect failed: certificate verify failed
this is output from redis log:
Error accepting a client connection: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
While I am using openssl client with same certificates, i am able to connect and get ping reply from Redis server
No matter if I change
tls-ca-cert-dir /etc/letsencrypt/live/myserver.net/
to
tls-ca-cert
on server side
or
--cacert /etc/letsencrypt/live/myserver.net/fullchain.pem to chain.pem on client side
I tried to all versions of
tls-protocols ""
and change
tls-auth-clients no
to
tls-auth-clients optional
but I am still stuck with same error
OpenSSL version is 1.1.1
Redis version is 6.0.8
OS: Ubuntu 20.04
Can you help me to find out reason why is TLS not working, please?
Thank you
Wil
Ahh, SOLVED!
I was putting wrong CA chain. I had to chain root and intermediate certs downloaded from LE website into new file. It may come handy for someone with same problem.
I have set up elasticache with redis and the host is rechable which I can confirm with telnet, when Redis commands are issued it does not return any result, either with ubuntu#ip-10-0-2-8:~$ redis-cli -h master.xxxxxx-xxxx.xxxxx.xxxx.cache.amazonaws.com -p 6379 INFO or and very unfortunately AWS cant show you redis logs
The redis-cli client does not support SSL/TLS connections. To use the
redis-cli to access an ElastiCache for Redis node (cluster mode
disabled) with in-transit encryption, you can use the stunnel package
in your Linux-based clients. The stunnel command can create an SSL
tunnel to Redis nodes specified in the stunnel configuration. After
the tunnel is established, the redis-cli can be used to connect an
in-transit encryption enabled cluster node.
Source: https://aws.amazon.com/premiumsupport/
So you can either use stunnel or disabling in-transit encryption.
You need to add firewall rule to allow other machine to access your redis server. I meant you need to enable firewall rule to allow 6379 port accessible from outside. Following article will will help you to do this.
Also please make sure redis is running on port 6379 or some other port.
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/accessing-elasticache.html#access-from-outside-aws
I am using elastic cache single node shard redis 4.0 later version.
I enabled In-Transit Encryption and gave redis auth token.
I created one bastion host with stunnal using this link
https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-connect-redis-node/
I am able to connect to elastic cache redis node using following way
redis-cli -h hostname -p 6379 -a mypassword
and i can do telnet also.
BUT
when I ping (expected response "PONG") on redis-cli after connection it is giving
"Error: Connection reset by peer "
I checked security group of both side.
Any idea ?
Bastion Host ubuntu 16.04 machine
As I mentioned in question, I was running the command like this:
redis-cli -h hostname -p 6379 -a mypassword
The correct way to connect into a ElastiCache cluster through stunnel should be using "localhost" as the host address,like this:
redis-cli -h localhost -p 6379 -a mypassword
There is explanation for using the localhost address:
when you create a tunnel between your bastion server and the ElastiCache host through stunnel, the program will start a service that listen to a local TCP port (6379), encapsulate the communication using the SSL protocol and transfer the data between the local server and the remote host.
you need to start the stunnel, check if the service is listening on the localhost address (127.0.0.1), and connect using the "localhost" as the destination address: "
Start stunnel. (Make sure you have installed stunnel using this link https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-connect-redis-node/)
$ sudo stunnel /etc/stunnel/redis-cli.conf
Use the netstat command to confirm that the tunnels have started:
$ netstat -tulnp | grep -i stunnel
You can now use the redis-cli to connect to the encrypted Redis node using the local endpoint of the tunnel:
$redis-cli -h localhost -p 6379 -a MySecretPassword
localhost:6379>set foo "bar"
OK
localhost:6379>get foo
"bar"
Most probably ElastiCache Redis Instance is using Encryption in-transit and Encryption at-rest and by design, the Redis CLI is not compatible with the encryption.
You need to setup stunnel to connect redis cluster
https://datanextsolutions.com/blog/how-to-fix-redis-cli-error-connection-reset-by-peer/
"Error: Connection reset by peer" indicates that Redis is killing your connection without sending any response.
One possible cause is you are trying to connect to the Redis node without using SSL, as your connection will get rejected by the Redis server without a response [1]. Make sure you are connecting through the correct port in your tunnel proxy. If you are connecting directly from the bastion host, you should be using local host.
Another option is that you have incorrectly configured your stunnel to not include a version of SSL that is supported by Redis. You should double check the config file is exactly the same as the one provided in the support doc.
It that doesn't solve your problem, you can try to build the cli included in AWS open source contribution.[2] You'll need to check out the repository, follow the instructions in the readme, and then do make BUILD_SSL=yes make redis-cli.
[1] https://github.com/madolson/redis/blob/unstable/src/ssl.c#L464
[2] https://github.com/madolson/redis/blob/unstable/SSL_README.md
From inside a docker container, I'm running
# openssl s_client -connect rubygems.org:443 -state -nbio 2>&1 | grep "^SSL"
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
That's all I get
I can't connect to any https site from within the docker container. The container is running on an openstack vm. The vm can connect via https.
Any advice?
UPDATE
root#ce239554761d:/# curl -vv https://google.com
* Rebuilt URL to: https://google.com/
* Hostname was NOT found in DNS cache
* Trying 216.58.217.46...
* Connected to google.com (216.58.217.46) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
and then it hangs.
Also, I'm getting intermittent successes now.
Sanity Checks:
changing the docker ips doesn't fix the problem
The docker containers work on my local machine
The docker containers work on other clouds
Docker 1.10.0 doesn't work in the vms
Docker 1.9.1 works in the vms
I was given a solution by the Docker community
OpenStack network seems to use lower MTU values and Docker does not infer the MTU settings from the host's network card since 1.10.
To run docker daemon with custom MTU settings, you can follow this blog post, that says:
$ cp /lib/systemd/system/docker.service /etc/systemd/system/docker.service
Edit a line in the new file to look like this:
ExecStart=/usr/bin/docker daemon -H fd:// --mtu=1454
Or (as suggested below by Dionysius), create and edit the file
/etc/systemd/system/docker.service.d/fixmtu.conf as follow:
[Service]
# Reset ExecStart & update mtu (see original command in /lib/systemd/system/docker.service)
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --mtu=1454.
MTU of 1454 is the value that seems to be common with OpenStack. You can look it up in your host using ifconfig.
Finally restart Docker:
$ sudo systemctl daemon-reload
$ sudo service docker restart