Configuring Kafka SSL with Let's Encrypt - ssl

I'm trying to setup a Kafka cluster and add SSL to it, but I'm getting always the same error:
INFO [SocketServer brokerId=0] Failed authentication with /XXX.XXX.XXX.XXX (SSL handshake failed) (org.apache.kafka.common.network.Selector)
I read several stackoverflow posts about this problem but I cannot fix it.
I'm using Kafka 2.5.0 on Ubuntu 18.04 with OpenJDK 14 and I used Let's Encrypt to generate certificates for my domain name (that are working perfectly with nginx).
First I created a pkcs12 file with the following command:
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out keystore.p12 -name kafka1 -CAfile chain.pem -caname root
...then I created the keystore (with the -ext option as described in kafka docs):
keytool -importkeystore -deststorepass 'STRONG_PASS' -destkeypass 'STRONG_PASS' -destkeystore keystore.jks -srckeystore keystore.p12 -srcstoretype PKCS12 -srcstorepass 'STRONG_PASS' -alias kafka1 -ext SAN=DNS:{FQDN}
...also I added the certificate to the truststore:
keytool -trustcacerts -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit -noprompt -importcert -file /etc/letsencrypt/live/YOURDOMAIN/chain.pem
At this point, I believe that I have all of the necessary steps to have the keystore and the truststore correctly configured.
My broker config is:
broker.id=0
listeners=SSL://mydomain.com:9092
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=/path/to/jdk/lib/security/cacerts
ssl.truststore.password=changeit
ssl.secure.random.implementation=SHA1PRNG
security.inter.broker.protocol=SSL
ssl.endpoint.identification.algorithm= # I tried with and without this and the problem persists
advertised.listeners=SSL://mydomain.com:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/path/to/kafka_data
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.num.partitions=3
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
min.insync.replicas=2
default.replication.factor=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=IP1:2181,IP2:2181,IP3:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=3000
Then I run the broker and try to execute the following command to know if everything is working:
openssl s_client -debug -connect mydomain.com:9092 -tls1 | head -n 50
And the error is displayed:
INFO [SocketServer brokerId=0] Failed authentication with /XXX.XXX.XXX.XXX (SSL handshake failed) (org.apache.kafka.common.network.Selector)
My domain is not the same as my hostname in the machine, I don't know if it could be a problem. I only want to add security to the first broker and then repeat the same in the other two, but first have the first one working.
What is wrong with my config? Maybe the truststore? or the hostname and domain name?

Related

How to run Wowza streaming engine over https

I'm trying to import my existing ssl certificates in my wowza streaming server.
I'm following this document: Import certificate
I'm able to convert my certificates in JKS format, and i have change VHost.xml file as suggested on the url. but when i restart server and check logs i found below error:
Here is my conf VHost.xml file: (i have replace password with asterisk just for screenshot)
Steps that i have follow:
sudo openssl pkcs12 -export -in /etc/ssl/private/nginx/certificate.crt -inkey /etc/ssl/private/nginx/private.key -name example.com -out example.com-PKCS-12.p12
sudo keytool -importkeystore -deststorepass ***** -destkeystore example.com.jks -srckeystore example.com-PKCS-12.p12 -srcstoretype PKCS12
sudo keytool -import -alias bundle -trustcacerts -file /etc/ssl/private/nginx/ca_bundle.crt -keystore /usr/local/WowzaStreamingEngine-4.7.6/conf/example.com.jks
After that i have open 443 port in VHost.xml and change keystore and password
Please help me to run wowza streaming engine over https server.
Thanks in advance

kafka 2 way ssl authentication

I am trying to setup 2 way ssl authentication. My requirement is broker should authenticate only specific clients.
My organization has a CA which issue all certificates in pkcs12 format. steps i followed are as follows.
get a certificate for the broker and configured it in the broker keystore
ssl.keystore.location=/home/kafka/certificate.p12
ssl.keystore.password=xxxxx
ssl.client.auth=required
get a certificate for the client and configured it in the client keystore
ssl.keystore.location=/home/kafka/certificate.p12
ssl.keystore.password=xxxxx
extracted the public certificate from the client certificate using keytool command
keytool -export -file cert -keystore certificate.p12 -alias "12345" -storetype pkcs12 -storepass xxxxx
imported the certificate into broker truststore. broker truststore contains only the client 12345 certificate.
keytool -keystore truststore.p12 -import -file cert -alias 12345 -storetype pkcs12 -storepass xxxxx -noprompt
configured the truststore in the broker.
ssl.truststore.location=/home/kafka/truststore.p12
ssl.truststore.password=xxxxx
configured the truststore in client. client truststore contains CA certificates.
ssl.truststore.location=/etc/pki/java/cacerts
ssl.truststore.password=xxxxx
When i run the broker and client i expect the broker to authenticate the client and establish ssl connection. but instead following error is thrown.
[2021-06-03 23:32:06,864] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (abc.com/10.129.140.212:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2021-06-03 23:32:06,866] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
I tried various things but nothing seems to work. when i replace the broker truststore with /etc/pki/java/cacerts truststore file which contains only the CA certificate
then it works fine. but it will authenticate any client which has certificate issued by the CA.
what could be the issue ?
The default format is jks,
use keytool to create a Java KeyStore (JKS) with the certificate and key for use by Kafka. You'll be prompted to create a new password for the resulting file as well as enter the password for the PKCS12 file from the previous step. Hang onto the new JKS password for use in configuration below.
$ keytool -importkeystore -srckeystore server.p12 -destkeystore kafka.server.keystore.jks -srcstoretype pkcs12 -alias myserver.internal.net
Note: It's safe to ignore the following warning from keytool.
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore server.p12 -destkeystore kafka.server.keystore.jks -srcstoretype pkcs12"

How to verify if keystore has all the certificates needed to connect to server?

I have been struggling with setting up our env on ssl. We get keystore files from our CA's.
It is fairly an automated process. However, we mostly struggle at the client side.
Currently, I run following command to check certs from server
openssl s_client -connect <server>:<port>
Once it prints the certs, I list keystores and verify DN, issuer, subject manully.
keytool -list -v -keystore keystore.jks
I would like to know if there is a command or any other way to feed the keystore.jks to openssl command and verify certs.
You can try following command:-
openssl s_client -verify 100 -showcerts -connect $HOST:$PORT -CAfile <(keytool -list -rfc -keystore truststore.jks -storepass changeit)
-keystore is the path of keystore.
-storepass is the password for the keystore
This will load your keystore as input to -CAfile
Command works as same as openssl s_client -connect $HOST:$PORT -CAfile <path-to-ca.pem>

How to install CA certificate on Nexus 3?

I have installed Nexus Repository Manager OSS 3.0.2-02 on my CentOS Linux release 7.4.1708.
Also I have CA certificate:
Issued to: \*.mycompany.com
Issued by: Go Daddy Secure Certificate Authority - G2
Valid from 2016-11-12 to 2018-01-11
The RSA private key match with my CA certificate, which was checked with Certificate Key Matcher and reported:
The certificate and private key match!
In Nexus3 directory I made the following changes:
Added to the file org.sonatype.nexus.cfg the lines:
nexus-args=${karaf.etc}/jetty.xml,${karaf.etc}/jetty-http.xml,${karaf.etc}/jetty-requestlog.xml,${karaf.etc}/jetty-https.xml
application-port-ssl=8443
Added to jetty-https.xml file next lines:
KeyStorePath /ssl/test.jks
KeyStorePassword 123456
KeyManagerPassword 123456
TrustStorePath ssl/test.jks
TrustStorePassword 123456
Created SSL directory in $NEXUS_HOME/etc/ and created inside the SSL directory a Java keystore file test.jks with command:
openssl pkcs12 -export -in mycompany.com.pem -inkey key.pem -name xxx.mycompany.com -out test.pks
keytool -importkeystore -deststorepass 123456 -destkeystore test.jks -srckeystore test.pks -srcstoretype PKCS12
keytool -import -alias bundle -trustcacerts -file gd_bundle.crt -keystore test.jks
After Nexus restart I can't not reach it by URL //xxx.mycompany.com:8443. Firefox says:
The page you are trying to view cannot be shown because the
authenticity of the received data could not be verified
Also trying to do: docker login xxx.mycompany.com:8443 and providing default admin:admin:123 credentials cause:
Error response from daemon: Get xxx.mycompany.com:8443/v1/users/:
x509: certificate signed by unknown authority
I also try to chain certificates by various scenarions from Google, including How to add certificate chain to keystore?, but got error on Docker:
Error response from daemon: Get //xxx.mycompany.com:8443/v1/users/: EOF
on Firefox, the same as first time and on Chrome:
//xxx.mycompany.com unexpectedly closed the connection
Question: Where is my mistake, or how to correctly install CA certificate on Nexus3?
I've found the resolution. Just needed to set in jetty-https.xml next lines:
<Set name="NeedClientAuth"><Property name="jetty.ssl.needClientAuth" default="false"/></Set>
<Set name="WantClientAuth"><Property name="jetty.ssl.wantClientAuth" default="false"/></Set>

Configure SSL on Jetty

I am trying to configure SSL on my Jetty.
I read this:
http://docs.codehaus.org/display/JETTY/How+to+configure+SSL
and created a key store.
Then, I jumped directly to section 4. But where is this configuration file I should configure Jetty?
I tried to serach for jetty.xml, but there is no such on my computer...
I had a lot of problems making it work but I finally foud out how to make it happend. I'm using ubuntu 10.04 with java 7. It may be possible to do it under windows but all the comands lines are bash commands, maybe possible to do the same with cigwin/mingw
I used Jetty 8.1.8. Download it from codehaus and choose the .tar.gz file for linux (.zip for windows).
Unzip the file in any directory you wish, this will be your {jetty} home folder for the sake of this article/answer.
Go to the {jetty}/etc directory.
Execute all the following command lines in order. Whenever a password is asked, input the same password all the time. The passwords are used to protect the key file, the key store and the certificate itself. Sometimes, a password will be asked to unlock the key store or to use a generated key. Once you will understand what everything is and how to use the passwords correctly, you may change those passwords when you feel ready (safer for production use). Otherwise, input the requested informations when asked.
openssl genrsa -des3 -out jetty.key
openssl req -new -x509 -key jetty.key -out jetty.crt
keytool -keystore keystore -import -alias jetty -file jetty.crt -trustcacerts
openssl req -new -key jetty.key -out jetty.csr
openssl pkcs12 -inkey jetty.key -in jetty.crt -export -out jetty.pkcs12
keytool -importkeystore -srckeystore jetty.pkcs12 -srcstoretype PKCS12 -destkeystore keystore
Now you have to edit {jetty}/etc/jetty-ssl.xml and configure your password to match the one you used during certificate generation. If you want to obfuscate your password, go back to the command line. Go tho your {jetty} home directory and execute the following:
java -cp lib/jetty-util-8.1.8.v20121106.jar org.eclipse.jetty.util.security.Password "{PASSWORD}"
Change {PASSWORD} for your actual password then past the obfuscated password, including the "OBF:" in all password fields found in jetty-ssl.xml. Note that a password obfuscated like that is hard to read for humans but easily unobfiscated programmatically. It just prevent developpers to know the password when they edit the file. All configuration files should be secured properly and their accesses be as restrictive as possible.
Edit {jetty}/start.ini and uncomment the line #etc/jetty-ssl.xml (just remove the #).
Start jetty:
java -jar start.jar
Now contact your server at: https://localhost:8443
Done!
Note that this answer is a quick way to enable SSL with jetty. To make it secure for production, you have to read some more on the subject.
Answer updated after more experience with keystores. I assure you this solution works perfectly with intermediate certificates (29/07/2015).
Note: PEM format means a readable file, certificates start with ---BEGIN CERTIFICATE--- and private keys start with -----BEGIN PRIVATE KEY----- line.
Here's an easy step by step guide. Start with an empty directory.
Skip to Step 2 if you have private key (PEM encoded .key)
Skip to Step 3 if you have certificate signing request (PEM encoded .csr)
Skip to Step 4 if you have your certificate (PEM encoded .crt or .pem)
Prepare (password-less) private key.
openssl genrsa -des3 -passout pass:1 -out domain.pass.key 2048
openssl rsa -passin pass:1 -in domain.pass.key -out domain.key
rm domain.pass.key
Prepare certificate signing request (CSR). We'll generate this using our key. Enter relevant information when asked. Note the use of -sha256, without it, modern browsers will generate a warning.
openssl req -key domain.key -sha256 -new -out domain.csr
Prepare certificate. Pick one:
a) Sign it yourself
openssl x509 -req -days 3650 -in domain.csr -signkey domain.key -out domain.crt
b) Send it to an authority
Your SSL provider will supply you with your certificate and their intermediate certificates in PEM format.
Add to trust chain and package it in PKCS12 format. First command sets a keystore password for convenience (else you'll need to enter password a dozen times). Set a different password for safety.
export PASS=LW33Lk714l9l8Iv
Pick one:
a) Self-signed certificate (no need for intermediate certificates)
openssl pkcs12 -export -in domain.crt -inkey domain.key -out domain.p12 -name domain -passout pass:$PASS
keytool -importkeystore -deststorepass $PASS -destkeypass $PASS -destkeystore domain.keystore -srckeystore domain.p12 -srcstoretype PKCS12 -srcstorepass $PASS -alias domain
b) Need to include intermediate certificates
Download intermediate certificates and concat them into one file. The order should be sub to root.
cat sub.class1.server.ca.pem ca.pem > ca_chain.pem
Use a -caname parameter for each intermediate certificate in chain file, respective to the order they were put into the chain file.
openssl pkcs12 -export -in domain.crt -inkey domain.key -out domain.p12 -name domain -passout pass:$PASS -CAfile ca_chain.pem -caname sub1 -caname root -chain
keytool -importkeystore -deststorepass $PASS -destkeypass $PASS -destkeystore domain.keystore -srckeystore domain.p12 -srcstoretype PKCS12 -srcstorepass $PASS -alias domain
Important note: Although keytool -list will only list one entry and not any intermediate certificates, it will work perfectly.
Configure jetty.
Move domain.keystore file to JETTY_HOME/etc/.
Pick one:
a) You're using new start.ini style configuration (Jetty 8+):
jetty.keystore=etc/domain.keystore
jetty.truststore=etc/domain.keystore
jetty.keystore.password=LW33Lk714l9l8Iv
jetty.keymanager.password=LW33Lk714l9l8Iv
jetty.truststore.password=LW33Lk714l9l8Iv
b) You're using old style configuration with .xml files (you should upgrade to new style!):
Edit JETTY_HOME/etc/jetty-ssl.xml file and change the part below. Replace password parts to match your password. We don't define KeyManagerPassword because our key has no password.
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<New id="sslContextFactory" class="org.eclipse.jetty.http.ssl.SslContextFactory">
<Set name="KeyStore"><Property name="jetty.home" default="." />/etc/keystore</Set>
<Set name="KeyStorePassword">LW33Lk714l9l8Iv</Set>
<Set name="TrustStore"><Property name="jetty.home" default="." />/etc/keystore</Set>
<Set name="TrustStorePassword">LW33Lk714l9l8Iv</Set>
</New>
<Call name="addConnector">...</Call>
</Configure>
Edit start.ini file to include jetty-ssl.xml file.
(Re)start jetty.
Note that this keystore file can also be used with other containers like Tomcat. Good luck!
A default configuration file for Jetty and is located at $JETTY_HOME/etc/jetty.xml
If you are using maven's jetty plugin you will need to specify ssl keystore details in your pom.xml file. See this question for details
Just bought a cert from godaddy for mere $6/year. Great deal while it lasts. Here are the steps I followed to set it up on Amazon EC2/Ubuntu/Jetty based on these sites and Jean-Philippe Gravel's answer.
http://docs.codehaus.org/display/JETTY/How+to+configure+SSL
http://community.xmatters.com/docs/DOC-1228#.UgWsI1MU7lc
keytool -keystore keystore -alias jettykey -genkey -keyalg RSA
Note that "First and last name" must be your FQDN (without http://). On my first attempt I had dutifully put my first and last name, but godaddy has good warnings and rejected it.
Generate a CSR file for Godaddy:
keytool -certreq -alias jetty -keystore keystore -file jetty.csr
Submit this in the Godaddy form to create the certificate, including the BEGIN/END "NEW CERTIFICATE REQUEST".
(Godaddy requires you to verify its your site. There a couple methods for this and since I bought the domain name via a proxy, I found it easiest and quickest to verify by hosting an html page generated by godaddy.)
Download the zip containing both certificate and intermediary certificate from godaddy. There is a list of server types to choose from. I choose "other". Then combine cert with intermediary cert.
cat mydomain.com.crt gd_bundle.crt > certchain.txt
export my private key
keytool -importkeystore -srckeystore keystore -destkeystore intermediate.p12 -deststoretype PKCS12
openssl pkcs12 -in intermediate.p12 -out jettykey.pem -nodes
combine private key and certificate
openssl pkcs12 -export -inkey jettykey.pem -in certchain.txt -out jetty.pkcs12
import pkcs12 cert (alias becomes 1)
keytool -importkeystore -srckeystore jetty.pkcs12 -srcstoretype PKCS12 -destkeystore keystore
(I backed up the keystore then deleted the original key. I did this while troubleshooting and this may or may not be required by Jetty.)
keytool -delete -keystore keystore -alias jettykey
sudo cp keystore /usr/share/jetty/etc/
sudo vi /usr/share/jetty/etc/jetty-ssl.xml
Modify your.store.password, your.key.password, and your.trust.password accordingly. If you want to obfuscate it, use
java -cp /usr/share/jetty/lib/jetty.jar:/usr/share/jetty/lib/jetty-util.jar org.mortbay.jetty.security.Password <your.password>
Indicate to Jetty to load the jetty-ssl.xml file.
sudo echo "/etc/jetty/jetty-ssl.xml" >> /etc/jetty/jetty.conf
sudo /sbin/iptables -t nat -I PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8443
(Also modify Amazon EC2 security group to allow 443)
sudo service jetty start
If you happen to work with Jetty 9.3 then you should change configuration in start.d/ssl.ini:
jetty.sslContext.keyStorePath=mystore.jks
jetty.sslContext.keyStorePassword=X
jetty.sslContext.keyManagerPassword=X
jetty.sslContext.trustStorePath=mystore.jks
jetty.sslContext.trustStorePassword=X
Where:
mystore.jks is your store generated with the keytool
X is your password in plain text (I would recommend skipping obfuscation as it only gives you false security)
The store is exactly the same as you would generate for Tomcat. Even if you used different Java version to generate the keystore that should not be a problem.
When trying on Windows with Jetty as Maven plugin the following steps can help:
pom.xml
<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>8.1.11.v20130520</version>
<configuration>
<scanIntervalSeconds>10</scanIntervalSeconds>
<webApp>
<contextPath>/yourappcontext</contextPath>
</webApp>
<connectors>
<connector implementation="org.eclipse.jetty.server.nio.SelectChannelConnector">
<port>9090</port>
<maxIdleTime>1</maxIdleTime>
</connector>
<connector implementation="org.eclipse.jetty.server.ssl.SslSocketConnector">
<port>9443</port>
<keystore>src/test/resources/keystore</keystore>
<keyPassword>123456</keyPassword>
<password>123456</password>
</connector>
</connectors>
</configuration>
</plugin>
Generate key/certificate using the JDK tool keytool:
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
This command will generate a file keystore which we need to put at the following (or what ever you like until it is configured in the keystore element) path src/test/resources/keystore.