SSL Certification Verify Failed on Heroku Redis - ssl

I'm deploying a flask app on Heroku using a Redis premium plan. I get the following error: 'SSL Certification Verify Failed'. Attempted fixes:
Downgrading to Redis 5
Passing ssl_cert_reqs=None to the Redis constructor in redis-py
A solution to this problem could be:
Explain how to disable TLS certification on heroku redis premium plans
Explain how to make TLS certification work on heroku redis premium plans
From Heroku's docs, this may be a hint: 'you must enable TLS in your Redis client’s configuration in order to connect to a Redis 6 database'. I don't understand what this means.

I solved my problem by adding ?ssl_cert_reqs=CERT_NONE to the end of REDIS_URL in my Heroku config.

You can disable TLS certification on Heroku by downgrading to Redis 5 and passing ssl_cert_reqs=None to the Redis constructor.
$ heroku addons:create heroku-redis:premium-0 --version 5
from redis import ConnectionPool, Redis
import os
connection_pool = ConnectionPool.from_url(os.environ.get('REDIS_URL'))
app.redis = Redis(connection_pool=connection_pool, ssl_cert_reqs=None)
My mistake was not doing both at the same time.
An ideal solution would explain how to configure TLS certification for Redis 6.

The docs are actually incorrect, you have to set SSL to verify_none because TLS happens automatically.
From Heroku support:
"Our data infrastructure uses self-signed certificates so certificates
can be cycled regularly... you need to set the verify_mode
configuration variable to OpenSSL::SSL::VERIFY_NONE"
I solved this by setting the ssl_params to verify_none:
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
For me it was where I config redis (in a sidekiq initializer):
# config/initializers/sidekiq.rb
Sidekiq.configure_client do |config|
config.redis = { url: ENV['REDIS_URL'], size: 1, network_timeout: 5,
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end
Sidekiq.configure_server do |config|
config.redis = { url: ENV['REDIS_URL'], size: 7, network_timeout: 5,
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end

On Heroku (assuming Heroku Redis addon), the redis TLS route already has the ssl_cert_reqs param sorted out. A common oversight that can cause errors in cases like this on heroku is: using REDIS_URL over REDIS_TLS_URL.
Solution:
redis_url = os.environ.get('REDIS_TLS_URL')

This solution works with redis 6 and python on Heroku
import os, redis
redis_url = os.getenv('REDIS_URL')
redis_store = redis.from_url(redis_url, ssl_cert_reqs=None)
In my local development environment I do not use redis with the rediss scheme, so I use a function like this to allow work in both cases:
def get_redis_store():
'''
Get a connection pool to redis based on the url configured
on env variable REDIS_URL
Returns
-------
redis.ConnectionPool
'''
redis_url = os.getenv('REDIS_URL')
if redis_url.startswith('rediss://'):
redis_store = redis.from_url(
redis_url, ssl_cert_reqs=None)
else:
redis_store = redis.from_url(redis_url)
return redis_store

If using the django-rq wrapper and trying to deal with this, be sure to not use the URL parameter with SSL_CERTS_REQS. There is an outstanding issue that describes this all, but basically you need to specify each connection param instead of using the URL.

Related

Heroku crashes after Heroku Redis upgrade from Hobby to Premium 0

Problem
When I upgraded from the Heroku Redis Hobby plan to the Heroku Redis Premium 0 plan, Heroku kept crashing with an H10 error.
Cause
Redis 6 requires TLS to connect. However, Heroku manages requests from the router level to the application level involving Self Signed Certs. Turns out, Heroku terminates SSL at the router level and requests are forwarded from there to the application via HTTP while everything is behind Heroku's Firewall and security measures.
Links that helped track down the cause:
https://ogirginc.github.io/en/heroku-redis-ssl-error
How to enable TLS for Redis 6 on Sidekiq?
Solution
Customize the options passed into Redis so that tls.rejectUnauthorized is set to false.
const Queue = require('bull');
const redisUrlParse = require('redis-url-parse');
const REDIS_URL = process.env.REDIS_URL || 'redis://127.0.0.1:6379';
const redisUrlParsed = redisUrlParse(REDIS_URL);
const { host, port, password } = redisUrlParsed;
const bullOptions = REDIS_URL.includes('rediss://')
? {
redis: {
port: Number(port),
host,
password,
tls: {
rejectUnauthorized: false,
},
},
}
: REDIS_URL;
const workQueue = new Queue('work', bullOptions);
Adding (on top of yeoman great answer) -
If you find yourself hitting SSL verification errors on Heroku when using django-rq and the latest Redis add-on -
Know that the RQ_QUEUES definition on django's settings.py supports SSL_CERT_REQS and you may specifically set it to None for solving these issues.
( inspired from https://paltman.com/how-to-turn-off-ssl-verify-django-rq-heroku-redis/ ).
Note that it requires boosting django-rq to a version >= 2.5.1.
That might be relevant for all users who apply Redis only for queuing (e.g. with RedisRQ) and not for caching.

CouchDB 1.6.1 SSL error on Raspbian

I am trying to setup SSL in CouchDB 1.6.1 on Raspbian and I get that :
** {badarg,[{ets,select_delete,
[undefined,[{{{undefined,'_','_'},'_'},[],[true]}]],
[]},
{ets,match_delete,2,[{file,"ets.erl"},{line,655}]},
{ssl_pkix_db,remove_certs,2,[{file,"ssl_pkix_db.erl"},{line,221}]},
{ssl_connection,terminate,3,
[{file,"ssl_connection.erl"},{line,934}]},
{tls_connection,terminate,3,
[{file,"tls_connection.erl"},{line,326}]},
{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,595}]},
{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,517}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}
My version of erlang is Erlang/OTP 17 [erts-6.2].
My local.ini file contains :
httpsd = {couch_httpd, start_link, [https]}
[ssl]
port = 6984
cert_file = /etc/couchdb/cert/couchdb.crt
key_file = /etc/couchdb/cert/couchdb.key
Http works fine. Any idea?
Cheers.
This looks like the crash fixed by this pull request, merged into Erlang/OTP 18.2.
To the best of my knowledge, the crash itself is harmless: it occurs during connection termination, because the code is trying to clean up something that never was set up. However, it might draw your attention from other errors happening before it, such as incorrect path to key / certificate files.

Connecting to Cassandra (2.1.0) over SSL from cqlsh

i have cassandra 2.1.0 running on Debian 7.6.0 and cqlsh running on the same machine. when i try to connect through cqlsh,
$/usr/local/cassandra-2.1.0/bin/cqlsh --ssl --debug
i get the following error message:
Using CQL driver: <module 'cassandra' from '/usr/local/cassandra-2.1.0/bin/../lib/cassandra-driver-internal-only-2.1.0.post.zip/cassandra-driver-2.1.0.post/cassandra/__init__.py'>
Connection error: ('Unable to connect to any servers', {'127.0.0.1': SSLError(0, '_ssl.c:340: error:00000000:lib(0):func(0):reason(0)')})
the details are as follows. pls. let me know how to resolve this issue. thanks in advance.
server side
as explained in (http://www.datastax.com/documentation/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html), i have generated a keystore and have modified cassandra.yaml as follows:
client_encryption_options:
enabled: true
keystore: /usr/local/cassandra-2.1.0/ssl/.keystore
keystore_password: ***********
i have exported the public key of the server.
client side
copied the public key exported from the previous step into ~/keys/cassandra_node0.cert.
modified ~/.cassandra/cqlshrc as follows:
[connection]
hostname = 127.0.0.1
port = 9042
factory = cqlshlib.ssl.ssl_transport_factory
[tracing]
max_trace_wait = 10.0
[ssl]
certfile = ~/keys/cassandra_node0.cert
validate = true
I had the same issue
Although you probably found the solution by now, but I think it can be beneficial to record the solution for other people.
I followed the documentation from here to create a .pem certificate.
My cqlshrc ssl configuration looks as follows
[ssl]
certfile = /ssl/cqlsh.pem
validate = false
That worked for me.
As with all ssl related topics in cassandra's documentation, this part isn't covered well enough.

SSLCaertBadFile error heroku curb

I have a rake task that pulls and parses JSON data over an SSL connection from an external API.
I use a gem that wraps this external API and have no problems running locally, but the task fails when run on heroku with #<Curl::Err::SSLCaertBadFile: Curl::Err::SSLCaertBadFile>
I installed the piggyback SSL add-on, hoping it might fix it, but no dice.
Any ideas?
UPDATE
I managed to fix it by disabling ssl verification on the curl request previously set by the following two fields:
request.ssl_verify_peer
request.ssl_verify_host
I don't know enough about SSL to know exactly why the error was caused by these settings in a heroku environment or what the implications of disabling this are, aside from reduced security.
It is a bad idea to disable certificate checking. See http://www.rubyinside.com/how-to-cure-nethttps-risky-default-https-behavior-4010.html, http://jamesgolick.com/2011/2/15/verify-none..html and associated references for more on that topic.
The issue is that your HTTP client doesn't know where to find the CA certificates bundle on heroku.
You don't mention what client you are using, but here is an example for using net/https on heroku:
require "net/https"
require "uri"
root_ca_path = "/etc/ssl/certs"
url = URI.parse "https://example.com"
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = (url.scheme == "https")
if (File.directory?(root_ca_path) && http.use_ssl?)
http.ca_path = root_ca_path
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.verify_depth = 5
end
request = Net::HTTP::Get.new(url.path)
response = http.request(request)
Here is an example using Faraday:
Faraday.new "https://example.com", ssl: { ca_path: "/etc/ssl/certs" }
Good luck.

2-way SSL with CherryPy

From CherryPy 3.0 and onwards, one-way SSL can be turned on simply by pointing to the server certificate and private key, like this:
import cherrypy
class HelloWorld(object):
def index(self):
return "Hello SSL World!"
index.exposed = True
cherrypy.server.ssl_certificate = "keys/server.crt"
cherrypy.server.ssl_private_key = "keys/server.crtkey"
cherrypy.quickstart(HelloWorld())
This enables clients to validate the server's authenticity. Does anyone know whether CherryPy supports 2-way ssl, e.g. where the server can also check client authenticity by validating a client certificate?
If yes, could anyone give an example how is that done? Or post a reference to an example?
It doesn't out of the box. You'd have to patch the wsgiserver to provide that feature. There is a ticket (and patches) in progress at http://www.cherrypy.org/ticket/1001.
I have been looking for the same thing. I know there are some patches on the CherryPy site.
I also found the following at CherryPy SSL Client Authentication. I haven't compared this vs the CherryPy patches but maybe the info will be helpful.
We recently needed to develop a quick
but resilient REST application and
found that CherryPy suited our needs
better than other Python networking
frameworks, like Twisted.
Unfortunately, its simplicity lacked a
key feature we needed, Server/Client
SSL certificate validation. Therefore
we spent a few hours writing a few
quick modifications to the current
release, 3.1.2. The following code
snippets are the modifications we
made:
cherrypy/_cpserver.py
## -55,7 +55,6 ## instance = None ssl_certificate = None ssl_private_key
= None
+ ssl_ca_certificate = None nodelay = True
def __init__(self):
cherrypy/wsgiserver/__init__.py
## -1480,6 +1480,7 ##
# Paths to certificate and private key files ssl_certificate = None ssl_private_key = None
+ ssl_ca_certificate = None
def __init__(self, bind_addr, wsgi_app, numthreads=10, server_name=None, max=-1, request_queue_size=5, timeout=10, shutdown_timeout=5):
## -1619,7 +1620,9 ##
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) if self.nodelay: self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
- if self.ssl_certificate and self.ssl_private_key:
+ if self.ssl_certificate and self.ssl_private_key and \
+ self.ssl_ca_certificate:
+ if SSL is None: raise ImportError("You must install pyOpenSSL to use HTTPS.")
## -1627,6 +1630,11 ## ctx = SSL.Context(SSL.SSLv23_METHOD) ctx.use_privatekey_file(self.ssl_private_key) ctx.use_certificate_file(self.ssl_certificate)
+ x509 = crypto.load_certificate(crypto.FILETYPE_PEM,
+ open(self.ssl_ca_certificate).read())
+ store = ctx.get_cert_store()
+ store.add_cert(x509)
+ ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT, lambda *x:True) self.socket = SSLConnection(ctx, self.socket) self.populate_ssl_environ()
The above patches require the
inclusion of a new configuration
option inside of the CherryPy server
configuration,
server.ssl_ca_certificate. This
option identifies the certificate
authority file that connecting clients
will be validated against, if the
client does not present a valid client
certificate it will close the
connection immediately.
Our solution has advantages and
disadvantages, the primary advantage
being if the connecting client doesn’t
present a valid certificate it’s
connection is immediately closed.
This is good for security concerns as
it does not permit the client any
access into the CherryPy application
stack. However, since the restriction
is done at the socket level the
CherryPy application can never see the
client connecting and hence the
solution is somewhat inflexible.
An optimal solution would allow the
client to connect to the CherryPy
socket and send the client certificate
up into the application stack. Then a
custom CherryPy Tool would validate
the certificate inside of the
application stack and close the
connection if necessary; unfortunately
because of the structure of CherryPy’s
pyOpenSSL implementation it is
difficult to retrieve the client
certificate inside of the application
stack.
Of course the patches above should
only be used at your own risk. If you
come up with a better solution please
let us know.
If the current version of CherryPy does not support client certificate verification, it is possible to configure CherryPy to listen to 127.0.0.1:80, install HAProxy to listen to 443 and verify client side certificates and to forward traffic to 127.0.0.1:80
HAProxy is simple, light, fast and reliable.
An example of HAProxy configuration