Problem
When I upgraded from the Heroku Redis Hobby plan to the Heroku Redis Premium 0 plan, Heroku kept crashing with an H10 error.
Cause
Redis 6 requires TLS to connect. However, Heroku manages requests from the router level to the application level involving Self Signed Certs. Turns out, Heroku terminates SSL at the router level and requests are forwarded from there to the application via HTTP while everything is behind Heroku's Firewall and security measures.
Links that helped track down the cause:
https://ogirginc.github.io/en/heroku-redis-ssl-error
How to enable TLS for Redis 6 on Sidekiq?
Solution
Customize the options passed into Redis so that tls.rejectUnauthorized is set to false.
const Queue = require('bull');
const redisUrlParse = require('redis-url-parse');
const REDIS_URL = process.env.REDIS_URL || 'redis://127.0.0.1:6379';
const redisUrlParsed = redisUrlParse(REDIS_URL);
const { host, port, password } = redisUrlParsed;
const bullOptions = REDIS_URL.includes('rediss://')
? {
redis: {
port: Number(port),
host,
password,
tls: {
rejectUnauthorized: false,
},
},
}
: REDIS_URL;
const workQueue = new Queue('work', bullOptions);
Adding (on top of yeoman great answer) -
If you find yourself hitting SSL verification errors on Heroku when using django-rq and the latest Redis add-on -
Know that the RQ_QUEUES definition on django's settings.py supports SSL_CERT_REQS and you may specifically set it to None for solving these issues.
( inspired from https://paltman.com/how-to-turn-off-ssl-verify-django-rq-heroku-redis/ ).
Note that it requires boosting django-rq to a version >= 2.5.1.
That might be relevant for all users who apply Redis only for queuing (e.g. with RedisRQ) and not for caching.
Related
I am trying to migrate a socket io service from GCP (App Engine) to a kubernetes cluster.
Everything works fine on the GCP side (we have one instance of the server without replicas).
The migration to k8s is going very well, except that when connecting the client socket to the server, it does not receive some information:
In transport 'polling': Of course, as there are two pods, this doesn't work properly anymore and the client socket keeps deconnecting / reconnecting in loop.
In 'websocket' transport: The connection is correctly established, the client can receive data from the server in 'broadcast to all client' mode => socket.emit('getDeviceList', os.hostname()) but, as soon as the server tries to send data only to the concerned client io.of(namespace).to(socket.id).emit('getDeviceList', JSON.stringify(obj)), this one doesn't receive anything...
Moreover, I modified my service to have only one pod for a test, the polling mode works correctly, but, I find myself in the same case as the websocket mode => I can't send an information to a precise client...
Of course, the same code on the App Engine side works correctly and the client receives everything correctly.
I'm working with:
"socket.io": "^3.1.0",
"socket.io-redis": "^5.2.0",
"vue": "^2.5.18",
"vue-socket.io": "3.0.7",
My server side configuration:
var io = require('socket.io')(server, {
pingTimeout: 5000,
pingInterval : 2000,
cors: {
origin: true,
methods: ["GET", "POST"],
transports: ['websocket', 'polling'],
credentials: true
},
allowEIO3: true
});
io.adapter(redis({ host: redis_host, port: redis_port }))
My front side configuration:
Vue.use(new VueSocketIO({
debug: true,
connection: 'path_to_the_socket_io/namespace,
options: {
query: `id=..._timestamp`,
transports: ['polling']
}
}));
My ingress side annotation:
kubernetes.io/ingress.class: nginx kubernetes.io/ingress.global-static-ip-name: ip-loadbalancer
meta.helm.sh/release-name: xxx
meta.helm.sh/release-namespace: xxx -release nginx.ingress.kubernetes.io/affinity: cookie nginx.ingress.kubernetes.io/affinity-mode: persistent nginx.ingress.kubernetes.io/force-ssl-redirect: true nginx.ingress.kubernetes.io/proxy-connect-timeout: 10800
nginx.ingress.kubernetes.io/proxy-read-timeout: 10800
nginx.ingress.kubernetes.io/proxy-send-timeout: 10800
nginx.org/websocket-services: app-sockets-cluster-ip-service
My question is : why i can get broadcast to all user message and not specific message to my socket ?
Can someone try to help me ? :)
Thanks a lot !
I found the solution in the day.and share it.
In fact, the problem is not due to the kubernetes Cluster but due to the socket io and socket io redis adapter version.
I was using socket.io: 3.x.x and using socket.io-redis: 5.x.x
In fact, i need to use the socket.io-redis: 6.x.x with this version of socket io :)
You can find the compatible version of socket io and redis adapter here:
https://github.com/socketio/socket.io-redis-adapter#compatibility-table
Thanks a lot.
I'm deploying a flask app on Heroku using a Redis premium plan. I get the following error: 'SSL Certification Verify Failed'. Attempted fixes:
Downgrading to Redis 5
Passing ssl_cert_reqs=None to the Redis constructor in redis-py
A solution to this problem could be:
Explain how to disable TLS certification on heroku redis premium plans
Explain how to make TLS certification work on heroku redis premium plans
From Heroku's docs, this may be a hint: 'you must enable TLS in your Redis client’s configuration in order to connect to a Redis 6 database'. I don't understand what this means.
I solved my problem by adding ?ssl_cert_reqs=CERT_NONE to the end of REDIS_URL in my Heroku config.
You can disable TLS certification on Heroku by downgrading to Redis 5 and passing ssl_cert_reqs=None to the Redis constructor.
$ heroku addons:create heroku-redis:premium-0 --version 5
from redis import ConnectionPool, Redis
import os
connection_pool = ConnectionPool.from_url(os.environ.get('REDIS_URL'))
app.redis = Redis(connection_pool=connection_pool, ssl_cert_reqs=None)
My mistake was not doing both at the same time.
An ideal solution would explain how to configure TLS certification for Redis 6.
The docs are actually incorrect, you have to set SSL to verify_none because TLS happens automatically.
From Heroku support:
"Our data infrastructure uses self-signed certificates so certificates
can be cycled regularly... you need to set the verify_mode
configuration variable to OpenSSL::SSL::VERIFY_NONE"
I solved this by setting the ssl_params to verify_none:
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
For me it was where I config redis (in a sidekiq initializer):
# config/initializers/sidekiq.rb
Sidekiq.configure_client do |config|
config.redis = { url: ENV['REDIS_URL'], size: 1, network_timeout: 5,
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end
Sidekiq.configure_server do |config|
config.redis = { url: ENV['REDIS_URL'], size: 7, network_timeout: 5,
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
end
On Heroku (assuming Heroku Redis addon), the redis TLS route already has the ssl_cert_reqs param sorted out. A common oversight that can cause errors in cases like this on heroku is: using REDIS_URL over REDIS_TLS_URL.
Solution:
redis_url = os.environ.get('REDIS_TLS_URL')
This solution works with redis 6 and python on Heroku
import os, redis
redis_url = os.getenv('REDIS_URL')
redis_store = redis.from_url(redis_url, ssl_cert_reqs=None)
In my local development environment I do not use redis with the rediss scheme, so I use a function like this to allow work in both cases:
def get_redis_store():
'''
Get a connection pool to redis based on the url configured
on env variable REDIS_URL
Returns
-------
redis.ConnectionPool
'''
redis_url = os.getenv('REDIS_URL')
if redis_url.startswith('rediss://'):
redis_store = redis.from_url(
redis_url, ssl_cert_reqs=None)
else:
redis_store = redis.from_url(redis_url)
return redis_store
If using the django-rq wrapper and trying to deal with this, be sure to not use the URL parameter with SSL_CERTS_REQS. There is an outstanding issue that describes this all, but basically you need to specify each connection param instead of using the URL.
Is there any node modules for sailsjs framework to make ssl certificate using let's encrypt?
There is a middleware that enables http->https redirect and also handles the ACME-validation requests from Let's Encrypt. As far as I can tell, it does not actually trigger the renewal, nor writes anything, but I believe that the ACME-scripts handle that as cron-jobs every 3 months or so, allowing you app to just validate automatically when they run. I haven't implemented this myself yet though.
I would also ask you to really consider using CloudFlare or some other SSL-termination service, as that also gives you a lot of other benefits like DDoS protection, some CDN-features etc.
Docs:#sailshq/lifejacket
As has been mentioned, you should consider the best overall solution in terms of CloudFlare or SSL-offload via nginx etc.
However, you can use greenlock-express.js for this to achieve SSL with LetsEncrypt directly within the Sails node environment.
The example below:
Configures an HTTP express app using greenlock on port 80 that handles the
redirects to HTTPS and the LetsEncrypt business logic.
Uses the greenlock SSL configuration to configure the primary Sails app as HTTPS on port 443.
Sample configuration for config/local.js:
// returns an instance of greenlock.js with additional helper methods
var glx = require('greenlock-express').create({
server: 'https://acme-v02.api.letsencrypt.org/directory'
, version: 'draft-11' // Let's Encrypt v2 (ACME v2)
, telemetry: true
, servername: 'domainname.com'
, configDir: '/tmp/acme/'
, email: 'myemail#somewhere.com'
, agreeTos: true
, communityMember: true
, approveDomains: [ 'domainname.com', 'www.domainname.com' ]
, debug: true
});
// handles acme-challenge and redirects to https
require('http').createServer(glx.middleware(require('redirect-https')())).listen(80, function () {
console.log("Listening for ACME http-01 challenges on", this.address());
});
module.exports = {
port: 443,
ssl: true,
http: {
serverOptions: glx.httpsOptions,
},
};
Refer to the greenlock documentation for fine-tuning configuration detail, but the above gets an out-of-the-box LetsEncrypt working with Sails.
Note also, that you may wish to place this configuration in somewhere like config/env/production.js as appropriate.
I’m trying to get SSL working in Meteor for https and websockets. I’m getting this error:
(STDERR) Error: listen EACCES 0.0.0.0:443
Here's my setup.
For SSL access I have installed nourharidy/meteor-ssl. I can use any comparable package or approach that people have found useful!
As required by nourharidy/meteor-ssl, in server/main.js I have:
SSL('/path/to/private/server.key','/path/to/private/server.crt', 443);
Here's the rest of my setup:
My ROOT_URL environment variable is:
https://10.0.1.10:443 //I’m using my Mac’s ip address so that another Mac can access it
In imports/startup/server/index.js I have:
//SET UP APOLLO QUERY / MUTATIONS / PUBSUB
const USING_HTTPS = true;
const httpProtocol = USING_HTTPS ? "https" : "http";
const localHostString = '10.0.1.10' //I’m using my Mac’s ip address so that another Mac can access it
const METEOR_PORT = 443;
const GRAPHQL_SUBSCRIPTION_PORT = 4000;
const subscriptionsEndpoint = `wss://${localHostString}:${GRAPHQL_SUBSCRIPTION_PORT}/subscriptions`;
const server = express();
server.use('*', cors({ origin: `${httpProtocol}://${localHostString}:${GRAPHQL_SUBSCRIPTION_PORT}` }));
server.use('/graphql', bodyParser.json(), graphqlExpress({
schema
}));
server.use('/graphiql', graphiqlExpress({
endpointURL: '/graphql',
subscriptionsEndpoint: subscriptionsEndpoint
}));
// We wrap the express server so that we can attach the WebSocket for subscriptions
const ws = createServer(server);
ws.listen(GRAPHQL_SUBSCRIPTION_PORT, () => {
console.log(`GraphQL Server is now running on ${httpProtocol}://${localHostString}:${GRAPHQL_SUBSCRIPTION_PORT}`);
// Set up the WebSocket for handling GraphQL subscriptions
new SubscriptionServer({
execute,
subscribe,
schema
}, {
server: ws,
path: '/subscriptions',
});
});
What am I missing?
Thanks very much in advance to all for any info!
The way Stack Overflow works is that you put some effort in, and get close to the problem, and we'll help you get through that last bit. Asking for help on how to use Google is what we call off-topic.
If you have done a Google search for Error: listen EACCES 0.0.0.0:443 would have yielded a bunch of results, the first of which is this:
Node.js EACCES error when listening on most ports
So I am willing to be helpful, but you need to help yourself too
Solved it! It turns out my server setup was fine.
Server-side I was building the WSS url like so:
var websocketUri = Meteor.absoluteUrl('subscriptions').replace(/http/, 'ws').replace('3100', '3200');
The Meteor docs for absoluteUrl say:
The server reads from the ROOT_URL environment variable to determine where it is running.
In Webstorm I have ROOT_URL set to https://10.0.nn.nn:3100. But for some reason absoluteUrl was returning http://10.0.nn.nn:3000. I.e. it had the wrong port.
After correcting the port-- it’s working!
UPDATE: I thought I had it solved when I posted a few days ago, but I was still missing something.
I'm now using ngrok to provide SSL to my dev system for use with HTTPS and WSS.
Here are a few details.
I run Meteor on localhost:3000.
I assign the value of "http://localhost:3000" to the ROOT_URL environment variable.
I run two simultaneous ngrok connections, one for HTTPS and one for WSS:
./ngrok http 3000
./ngrok http 3200
This gives me two ngrok urls, for example:
Forwarding https://9b785bd3.ngrok.io -> localhost:3000
Forwarding https://ec3d8027.ngrok.io -> localhost:3200
I access my app via the ngrok url, for example:
https://9b785bd3.ngrok.io
I build my HTTPS links and WSS links based on the ngrok addresses. For example:
const wssEndpoint = 'wss://ec3d8027.ngrok.io/subscriptions';
I hope this is helpful to others looking into this.
I have a rake task that pulls and parses JSON data over an SSL connection from an external API.
I use a gem that wraps this external API and have no problems running locally, but the task fails when run on heroku with #<Curl::Err::SSLCaertBadFile: Curl::Err::SSLCaertBadFile>
I installed the piggyback SSL add-on, hoping it might fix it, but no dice.
Any ideas?
UPDATE
I managed to fix it by disabling ssl verification on the curl request previously set by the following two fields:
request.ssl_verify_peer
request.ssl_verify_host
I don't know enough about SSL to know exactly why the error was caused by these settings in a heroku environment or what the implications of disabling this are, aside from reduced security.
It is a bad idea to disable certificate checking. See http://www.rubyinside.com/how-to-cure-nethttps-risky-default-https-behavior-4010.html, http://jamesgolick.com/2011/2/15/verify-none..html and associated references for more on that topic.
The issue is that your HTTP client doesn't know where to find the CA certificates bundle on heroku.
You don't mention what client you are using, but here is an example for using net/https on heroku:
require "net/https"
require "uri"
root_ca_path = "/etc/ssl/certs"
url = URI.parse "https://example.com"
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = (url.scheme == "https")
if (File.directory?(root_ca_path) && http.use_ssl?)
http.ca_path = root_ca_path
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.verify_depth = 5
end
request = Net::HTTP::Get.new(url.path)
response = http.request(request)
Here is an example using Faraday:
Faraday.new "https://example.com", ssl: { ca_path: "/etc/ssl/certs" }
Good luck.