I am using Airflow 1.7.0 with a LocalExecutor and documentation suggests that to enable SSL, we need to pass cert and key path and change the port to 443 as below
[webserver]
web_server_ssl_cert = <path to cert>
web_server_ssl_key = <path to key>
# Optionally, set the server to listen on the standard SSL port.
web_server_port = 443
base_url = http://<hostname or IP>:443
I have created cert and key generated using OpenSSL. The details supplied while creating the cert/key too are right.
However, the Airflow UI is still http and not https.
Any pointers would help!
Thank you!
Solved in this question How to enable SSL on Airflow Webserver? and answer https://stackoverflow.com/a/56760375/512111.
In short: generate a key, crt pair with
openssl req \
-newkey rsa:2048 -nodes -keyout domain.key \
-x509 -days 365 -out airflow.crt
and set in airflow.cfg like
web_server_ssl_cert = /path/to/airflow.crt
web_server_ssl_key = /path/to/airflow.key
Leave the webserver port unchanged. Restart the airflow webserver, go to https://hostname:port et voilà.
Airflow 1.7.0 doesn't support SSL. I just checked the webserver code of airflow 1.7.0. The code is given below. This function just starts the flask/gunicorn application on HTTP with the host and port. If you provide the certificate and mention the port as 443, it will simply start the application on http://<host>:443. It doesn't accept the SSL key and certificate. The webserver function of Airflow 1.7.0 is given below.
SSL feature is available with the latest version of the Apache Airflow. Please use the latest version for SSL support.
def webserver(args):
print(settings.HEADER)
from airflow.www.app import cached_app
app = cached_app(configuration)
workers = args.workers or configuration.get('webserver', 'workers')
if args.debug:
print(
"Starting the web server on port {0} and host {1}.".format(
args.port, args.hostname))
app.run(debug=True, port=args.port, host=args.hostname)
else:
print(
'Running the Gunicorn server with {workers} {args.workerclass}'
'workers on host {args.hostname} and port '
'{args.port}...'.format(**locals()))
sp = subprocess.Popen([
'gunicorn', '-w', str(args.workers), '-k', str(args.workerclass),
'-t', '120', '-b', args.hostname + ':' + str(args.port),
'airflow.www.app:cached_app()'])
sp.wait()
Go to AIRFLOW_HOME -> airflow.cfg. It has section named [webserver], under that there are two config properties like below:
web_server_ssl_cert =
web_server_ssl_key =
if there is no value like above means Airflow webserver is running on http (without certificate).
To enable SSL, use .p12 certificate (one you must have ordered) and use openssl to extract certificate and private key from .p12 file. openssl mostly ships with Linux so you can directly run on linux terminal.
Step1: Extract certificate using below command
openssl pkcs12 –in /path/cert.p12 -nokeys -clcerts –out /path/mycert.crt
Step2: Extract key using below command
openssl pkcs12 –in /path/cert.p12 -nocerts –out /path/mykey.key
Step3: Once certificate and key is generated, update airflow.cfg for
web_server_ssl_cert and web_server_ssl_key. Restart Airflow webserver.. are you
are done. Browse Airflow UI with https.
Related
I have successfully created a graylog server (in docker container) that ingests logs from filebeat on a separate machine.
However I of course would like to have the messages encrypted. I am attempting to set this up however I cannot seem to get graylog to accept the connection, instead it is always being reset by peer:
{"log.level":"error","#timestamp":"2023-01-04T15:08:57.746+0100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(async(tcp://<graylog_ip>:5044)): read tcp 192.168.178.99:54372-\><graylog_ip>:5044: read: connection reset by peer","service.name":"filebeat","ecs.version":"1.6.0"}
(Without tls the connection works as intended, a new line appears in graylog every time one is added to my test log file.)
Setup Details
I created a filebeat.crt and filebeat.key file with openssl. I confirmed that the hostname for the certificate was the same hostname for the server with graylog on it:
openssl genrsa -out filebeat.key 2048
openssl req -new -x509 -key filebeat.key -out filebeat.crt -days 3650
From my knowledge, a CA should not be required since I have copied the key myself, filebeat can just encrypt the data it sends with the filebeat.crt, then the server can decrypt with filebeat.key (perhaps this is not correct of me to imagine?)
I then copied both files to the server and local machine. In my compose file I mounted the key into the graylog container and restarted. Then I set up the input configuration that was working previously to have:
bind_address: 0.0.0.0
charset_name: UTF-8
no_beats_prefix: false
number_worker_threads: 12
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: /etc/graylog/server/filebeat.crt
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: true
tls_key_file: /etc/graylog/server/filebeat.key
tls_key_password: ********
Then in filebeat I have the following configuration (I also tried converting and using filebeat.pem for the certificate, but no change):
output.logstash:
hosts: ["<graylog_ip>:5044"]
ssl.certificate: '/etc/pki/tls/certs/filebeat.crt'
ssl.key: '/etc/pki/tls/private/filebeat.key'
I really cannot see the issue, any help would be greatly appreciated!
First, try to debug filebeat using
/usr/bin/filebeat -e -d '*' -c filebeat_5.1.2.conf
Probably you will discover that CA is needed or something like that.
But my best quess is that filebeat tries to verify hostname and certivicate name, your generated certificate could not have CN identical to hostname.
Proper solution is using:
ssl.verification_mode: none
Well, this solution works for me.
I have a Debian GCP instance that I'm trying to run a Python gRPC server. My instance has a static IP and I'm trying to establish a secure channel between my remote instance (server) and a local client.
I have generated self-signed OpenSSL certificates on the server and I am using the same certificates on the client. To generate I've used:
openssl req -newkey rsa:2048 -nodes -keyout ML.key -x509 -days 365 -out ML.crt
My server is set up like so (the .key and .crt files are loaded with an open as 'rb'):
server_credentials = grpc.ssl_server_credentials(((private_key, certificate_chain,),))
self.server.add_secure_port('0.0.0.0:%d' % self.port, server_credentials)
self.server.start()
My client is set up as:
host = '78.673.121.16' #this is the instance's static IP
port = 9063
certificate_chain = __load_ssl_certificate() #this loads the certificate file
# create credentials
credentials = grpc.ssl_channel_credentials(root_certificates=certificate_chain)
# create channel using ssl credentials
channel = grpc.secure_channel('{}:{}'.format(host, port), credentials)
and then I proceed to make a request.
At the server I am met with the following error, in response to my request:
E1017 17:21:22.964227087 1881 ssl_transport_security.cc:1807] No match found for server name: 78.673.121.16.
I have tried to change the Common Name (CN) of the certificate to localhost, 0.0.0.0 and 78.673.121.16 but to no avail.
Is there any suggestion?
I just had a similar problem, and was able to get it resolved finally. In my case I was hosting the server in a kubernetes cluster with a static ip and port. The key components of the solution were (in the server certificate):
Use the static IP address as the Common Name
Add the static IP address as a DNSName within the SubjectAlternativeName extension of the certificate
Step 2 turned out to be critical. In python (using grpc version 1.34.0) this was accomplished by:
from cryptography import x509
host = '78.673.121.16'
builder = x509.CertificateBuilder()
...
builder = builder.add_extension(x509.SubjectAlternativeName([x509.DNSName(host)]), critical=False)
try passing these options in secure_channel function call
options = {
'grpc.ssl_target_name_override' : 'localhost',
'grpc.default_authority': 'localhost'
}
channel = grpc.secure_channel('{}:{}'.format(host, port), credentials, options)
I have failed to find how to solve this and have opted to set up a permanent DNS for my instance instead. I was using GCP which, as of the time of writing, doesn't staightforwardly provide a way to assign this to an instance.
I switched to Azure, assigned the DNS to my instance and used that DNS and CN on my self-signed SSL certificate.
After that I changed the client (the server remains as originally) as:
host = 'myinstance.westus.azure.com' #this is the instance's DNS
port = 9063
This resolved my issue.
I am trying to setup gitlab on my server with SSL support. I am using a different port for the gitlab as follows in my nginx settings:
external_url 'https://myserver.com:2440
nginx['redirect_http_to_https'] = true
I also setup the SSL certificates as:
nginx['ssl_client_certificate'] = '/etc/ssl/AlphaSSLroot.crt'
nginx['ssl_certificate'] = '/etc/ssl/org.crt'
nginx['ssl_certificate_key'] = '/etc/ssl/org.key'
However, when I try to connect to my gitlab installation as https://myserver.com:2440, it comes back with the server refused to connect error. If I configure using http, it works.
I wonder if there is anything else I need to do to enable SSH here. I have my main website running on an apache web server using the same SSL certificate but on a completely different port (8080). So, I think I should be able to use the certificates.
It turns out that this was because of the passphrase and I had to remove that using
openssl rsa -in www.key -out new.key
and use the new.key in nginx.
I'm trying to renew my SSL certificate but there is some problem i'm probably missing. after i'v done the following steps the server keep using the old certificate and i do'nt know why.
here'w what i have done:
Create new csr file (domain.csr) + key file (domain.key)
openssl req -new -newkey rsa:2048 -nodes -keyout domain.key -out domain.csr
Copy csr file content and paste it to my ssl provider + get approval.
get 5 files from them and upload them to the server (domain.der,domain.pem
,domain.cer, chain.cer , domain.p7b )
set on apache ssl.conf file ,
SSLCertificateFile (domain.cer) SSLCertificateKeyFile (domain.key).
restart apache
for some reason my server is still using my old certificate.
is the something i'm doing wrong?
Well you figured it out yourself but in case anyone else is in same situation, here's some of the things you can check.
First up check locally whether this works, by running the following openssl command on the server (a crucial step we skipped!):
openssl s_client -connect localhost:443
This will show the cert presented to the client from Apache. If that's not the right one, then you know Apache config is at fault. If it is the right one then something downstream is a problem.
In your case you terminate SSL at the load balancer and forgot to change the cert there. Another issue could be browser caching the SSL cert (restart it, Ctrl+F5 to force refresh or better yet try another browser or a third party website like ssllabs.com).
Assuming it's a problem with Apache then you need to check the config to check all instances of the cert have been replace. The below command will show all the vhosts and what config they are configured in:
/usr/local/apache2/bin/apachectl -S
Alternatively just use standard find and grep unix commands to search your Apache config for the old or new cert:
find /usr/local/apache2/conf -name "*.conf" -exec grep olddomain.cer {} \; -print
Both those commands assume apache is installed in /usr/local/apache2 but change the path as appropriate.
If all looks good and you've definitely restarted Apache then you can try a full stop and restart as I have noticed sometimes a graceful restart of Apache doesn't always pick up new config. Before starting the web server back up again, check you can't connect from your browser (to ensure you're connecting to the server you think you're connecting to) and that the process is down with the following command:
ps -ef | grep httpd
and then finally start.
Another thing to check is that the cert you are installing is the one you think it is, using this openssl command to print out the cert details (assuming the cert is in x509 format but there are similar commands for other formats):
openssl x509 -in domain.cer -text
And last but not least check the Apache log files to see if any errors in there. Though would expect that to mean no cert is loaded rather than just the old one.
Good answer from #Barry.
Another aspect is apache is not the front most web server. From this conversation. It is possible that there are other web servers in front of apache.
Something like - nginx. In our case it was AWS ELB. We had to change cert in ELB in order to change.
We had a similiar problem to what #Akshay is saying above.
In order for the server to update the certificates we had to run some commands for Google Cloud Compute Engine Load Balancer:
gcloud compute target-https-proxies update
Hope this helps someone that is using GCP for hosting apache.
I am trying to host the MSOpenTech Redis instance securely. Looking at the base Redis Documentation I see a reference to using an SSL proxy on a linux box. Is there a windows equivalent or is the intended usage pattern to limit access via network constraints?
I was able to accomplish this using stunnel to perform the SSL offloading. I had to convert my server pfx to a pem using openssl and place it in the conf directory of the stunnel install. To keep it clean I move the port redis is listening on to 6380 so the connection topology is client ---(TLS 1.2) --> server:6379 --> Localhost:6380.
StackExchange.Redis Client Initialization
ConfigurationOptions options = new ConfigurationOptions
{
options.Ssl = true;
Password = "foobared"
};
options.EndPoints.Add("localhost:6379");
Stunnel Configuration
[redis-proxy]
sslVersion = all
options = NO_SSLv2
options = NO_SSLv3
cert = stunnel.pem
accept = 6379
connect = 6380
Openssl Command
openssl pkcs12 -in myservercert.pfx -out stunnel.pem -nodes