Istio remote error: tls: error decrypting message - ssl

I am starting out with Istio and trying to enable TLS on north-south traffic by creating a gateway resource enabled with TLS and am following this doco https://istio.io/docs/tasks/traffic-management/ingress/secure-ingress-mount/.
I have following everything to the dot but I keep getting this error from the Istiod pod logs:
2020-05-21T04:41:44.467181Z info grpc: Server.Serve failed to complete security handshake from "10.x.x.x:34774": remote error: tls: bad certificate
2020-05-21T04:41:54.416502Z info grpc: Server.Serve failed to complete security handshake from "10.x.x.x:56768": remote error: tls: error decrypting message
2020-05-21T04:42:00.305269Z info grpc: Server.Serve failed to complete security handshake from "10.x.x.x:56834": remote error: tls: error decrypting message
Any idea why this is happening? I did check for typos while creating certs but cannot find any.
This works for when I disable TLS and use HTTP. So I am assuming that the error is from using the certificates and the logs tell the same thing too.
Details about the cluster:
AWS EKS Version: 1.14
Istio Version: 1.51
Any help would be greatly appreciated!

Related

Kubevirt virtctl image-upload gives "remote error: tls: bad certificate error"

I am trying to upload windows10 image to pvc inorder to create a windows10 vm using kubevirt.
I used below virtctl command:
$ virtctl image-upload --image-path=/Win10_20H2_v2_English_x64.iso --pvc-name=win10-vm --access-mode=ReadWriteMany --pvc-size=5G --uploadproxy-url=https://<cdi-uploadproxy IP>:443 --insecure
Result :
pvc is created
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
win10-vm Bound pv0002 10Gi RWX 145m
win10-vm-scratch Bound pv0003 10Gi RWX 145m
cdi-image-upload pod is created.
[root#master kubevirt]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cdi cdi-apiserver-847d4bc7dc-l6fz7 1/1 Running 1 135m
cdi cdi-deployment-66d7555b79-d57bm 1/1 Running 1 135m
cdi cdi-operator-895bb5c74-hpk44 1/1 Running 1 135m
cdi cdi-uploadproxy-6c8698cd8b-z67xc 1/1 Running 1 134m
default cdi-upload-win10-vm 1/1 Running 0 53s
But upload gets timeout.When I checked logs of the cdi-upload-win10-vm pod,I got following errors:
I0413 10:58:38.695097 1 uploadserver.go:70] Upload destination: /data/disk.img
I0413 10:58:38.695263 1 uploadserver.go:72] Running server on 0.0.0.0:8443
2021/04/13 10:58:40 http: TLS handshake error from [::1]:57710: remote error: tls: bad certificate
2021/04/13 10:58:45 http: TLS handshake error from [::1]:57770: remote error: tls: bad certificate
2021/04/13 10:58:50 http: TLS handshake error from [::1]:57882: remote error: tls: bad certificate
2021/04/13 10:58:55 http: TLS handshake error from [::1]:57940: remote error: tls: bad certificate
2021/04/13 10:59:00 http: TLS handshake error from [::1]:58008: remote error: tls: bad certificate
2021/04/13 10:59:05 http: TLS handshake error from [::1]:58066: remote error: tls: bad certificate
2021/04/13 10:59:10 http: TLS handshake error from [::1]:58136: remote error: tls: bad certificate

Kafka SSL handshake failed issue

I am trying to enable SSL Authentication on my Kafka server. I am following 7.2 section in the Kafka documentation.
Followed all steps, but while calling the producer.bat file to send data in to the topic i get below error.
ERROR [Producer clientId=console-producer] Connection to node -1 failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
Caused by: java.security.cert.CertificateException: No name matching localhost found
I did create the certificate with CN=localhost.
The server host name verification may be disabled by setting ssl.endpoint.identification.algorithm to an empty string on the client.
Just set
ssl.endpoint.identification.algorithm=
It can help you.
I.e with an empty value:
ssl.endpoint.identification.algorithm=
For me it is an issue with input given to first name and last name for Keytool
Generally java.security.cert.CertificateException: No name matching localhost found means that the hostname in the certificate does not match the hostname of the server.
There is a great explanation of this error here: CertificateException: No name matching ssl.someUrl.de found
We encounterd the following errors, this might because the upgrade of Kafka's version from 1.x to 2.x.
javax.net.ssl.SSLHandshakeException: General SSLEngine problem ... javax.net.ssl.SSLHandshakeException: General SSLEngine problem ... java.security.cert.CertificateException: No name matching *** found
or
[Producer clientId=producer-1] Connection to node -2 failed authentication due to: SSL handshake failed
The default value for ssl.endpoint.identification.algorithm was changed to https, which performs hostname verification (man-in-the-middle attacks are possible otherwise). Set ssl.endpoint.identification.algorithm to an empty string to restore the previous behaviour. Apache Kafka Notable changes in 2.0.0
Solution:
SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, ""
When using Strimzi Kafka operator, use the cluster ca cert (add to jks truststore) to avoid this error. The client CA certificate was always throwing this error for me.
I got this error because of connection problems. Cause by me firewall.
The connection can be checked with:
openssl s_client -debug -connect servername:port -tls1_2
Answer should be "Verify return code: 0 (ok)
Other return codes could suggest that you have no access.

How to run remote code as user with certificate from a worker node

I created a user in the Master.
First I created a key and certificate for him: dan.key and dan.crt
Then I created it inside Kubernetes:
kubectl config set-credentials dan \
--client-certificate=/tmp/dan.crt \
--client-key=/tmp/dan.key
This is the ~/.kube/config:
users:
- name: dan
user:
as-user-extra: {}
client-certificate: /tmp/dan.crt
client-key: /tmp/dan.key
I want to be able to run commands from a remote worker as the user I created.
I know how to do it with service account token:
kubectl --server=https://192.168.0.13:6443 --insecure-skip-tls-verify=true --token="<service_account_token>" get pods
I copied the certifiacte and the key to the remote worker and ran:
[workernode tmp]$ kubectl --server=https://192.168.0.13:6443 --client-certificate=/tmp/dan.crt --client-key=/tmp/dan.key get pods
Unable to connect to the server: x509: certificate signed by unknown authority
I followed this question:
kubectl unable to connect to server: x509: certificate signed by unknown authority
I tried like he wrote:
kubectl proxy --address 0.0.0.0 --accept-hosts '.*'
But I am still receiving:
Unable to connect to the server: x509: certificate signed by unknown authority
I copied the certifiacte and the key to the remote worker and ran:
[workernode tmp]$ kubectl --server=https://192.168.0.13:6443 --client-certificate=/tmp/dan.crt --client-key=/tmp/dan.key get pods
Unable to connect to the server: x509: certificate signed by unknown authority
You were missing the critical piece of data telling kubectl how to trust the https: part of that request, namely --certificate-authority=/path/to/kubernetes/ca.pem
You didn't encounter that error while using --token=... because of the --insecure-skip-tls-verify=true which you should definitely, definitely not do.
I tried like he wrote:
kubectl proxy --address 0.0.0.0 --accept-hosts '.*'
But I am still receiving:
You have followed the wrong piece of advice from whatever article you were reading; that --accept-hosts flag only controls the remote hostnames from which kubectl proxy will accept connections, and has zero to do with SSL anythings.

cmder SSL verification - how to add to local store?

I am trying to install Drupal Vm via Cmder.
When I run 'Vagrant up;' command I get the following error:
Installing plugin vagrant-vbguest
**ERROR: SSL verification error at depth 1: unable to get local issuer certificate (20)
ERROR: You must add /C=US/ST=California/L=San Jose/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Root CA/emailAddress=support#zscaler.com to your local trusted store**
Vagrant failed to load a configured plugin source. This can be caused
by a variety of issues including: transient connectivity issues, proxy
filtering rejecting access to a configured plugin source, or a configured
plugin source not responding correctly. Please review the error message
below to help resolve the issue:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
Source: https://rubygems.org/
How to add "/C=US/ST=California/L=San Jose/O=Zscaler Inc./OU=Zscaler Inc./CN=Zscaler Root CA/emailAddress=support#zscaler.com" to my local trusted store ? Any help ?

nginx - log SSL handshake failures

I'm running an nginx server with SSL enabled.
My protocol / cipher settings are fairly secure, and I've checked them at ssllabs.com, but --
-- since this is a web service which is called by http clients that I have no control over, I have concerns about compatibility.
To the point:
Is there a way to log SSL handshake failures as they happen (if they happen) in my nginx logs?
For example, I've got SSLv3 disabled, and if I try to "curl -3" (forcing SSlv3) to my server, then I get this:
NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP)
Cannot communicate securely with peer: no common encryption algorithm(s).
Closing connection 0 curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).
I would like to log this type of error in server logs too, with the default nginx settings, there is nothing.
Enabling "debug" log level for the error log does what I want, will log SSL handshake errors -- but unfortunately it also logs too much other stuff, making the log too bloated, drowning out other potentially useful info.
You can use the info log level.