I'm developing a web app, and the app uses navigator.mediaDevices.getUserMedia, so it needs HTTPS.
So I created a dummy certificate and localhost KTOR loads it.
Which is great! Except it litters the logs with
2022-10-05 10:26:10.046 [eventLoopGroupProxy-3-4] WARN i.n.h.s.ApplicationProtocolNegotiationHandler - [id: 0x96c37772, L:0.0.0.0/0.0.0.0:8443] TLS handshake failed:
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
Is there any way to keep those from spamming the log?
(either through an improved self-signed cert, or through a filter, or a ktor config of "I get it, stop bothering me...")
I have deployed Kafka strimzi to my Kubernetes cluster.
My goal is to be able to connect to Kafka topic over SSL listener(port 9094) and be able to produce/consume messages.
Symptoms
[2022-05-05 10:20:11,252] ERROR Error when sending message to topic public.kafka.incoming.events with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Topic public.kafka.incoming.events not present in metadata after 60000 ms.
The message is not effectively sent by the producer or received by the consumer.
logs of strimzi-cluster-operator pod when selecting loadbalancer or ingress as type (i.e.: kafka.listeners.external.type).
failed to reconcile ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.my-kafka-project.svc:2181. Connection was not ready in 300000 ms.
Not sure if this is related with my Kubernetes cluster configuration problem. When I select NodePort - it works as expected and kafka + zookeper are started normally.
To make this story short, I have:
1. Activated the debugging flag:
log4j.logger.org.apache.kafka=INFO -> log4j.logger.org.apache.kafka=DEBUG
and I am able to read the logs from Kafka server.
2. Configured/generated the signed certificate, keystore/truststore JKs and the configuration I use looks like:
cat <<EOT > ssl.properties
security.protocol=SSL
ssl.protocol=TLSv1.2
bootstrap.servers=my-cluster-kafka-external-bootstrap:9094
ssl.truststore.location=/home/kafka/user-truststore.jks
ssl.truststore.password=<password>
ssl.keystore.location=/home/kafka/user-keystore.jks
ssl.keystore.password=<password>
ssl.key.password=<password>
ssl.endpoint.identification.algorithm=
EOT
3. Distinctions between case #1 which works and case #2 which does not work and is why I am asking this question.
Case 1: This is the case which works normally, I am connecting to Kafka topics from within the pod which is running Kafka server.
Case 2: In the second case, which is not working as expected, the connecting goes to a load-balancer(i.e which is exposed to the internet).
The load-balancer connection is then established through a private network (10.20.10.0/24) which communicates with the Kubernetes cluster.
Case 1 (which works, and is why I have put here to help solve this problem)
Testing from a Kafka server pod (within the cluster).
Producer
[kafka#my-cluster-kafka-0 ~]$ /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-external-bootstrap:9094 --topic public.kafka.incoming.events --producer.config consumer-ssl.properties
>ttesttt
>
Consumer
[kafka#my-cluster-kafka-0 ~]$ /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-0.my-cluster-kafka-brokers.my-kafka-project.svc.cluster.local:9094 --topic public.kafka.incoming.events --consumer.config consumer-ssl.properties
ttesttt
Confirmation
Confirmation based on the Kafka server logs.
When a Producer is connecting successfully
2022-05-05 13:40:07,988 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-xx.xxx.xx4.xx7:45150-42 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/xx.xxx.xx4.xx7:45150], selector=sun.nio.ch.EPollSelectorImpl#5c2628ee, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost 'xx.xxx.xx4.xx7' peerPort 45150 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
2022-05-05 13:40:07,989 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /xx.xxx.xx4.xx7 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
When a Consumer is connecting successfully
2022-05-05 13:43:56,053 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-10.15.19.76:40094-43 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/10.15.19.76:40094], selector=sun.nio.ch.EPollSelectorImpl#7df0ddcc, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost '10.15.19.76' peerPort 40094 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
2022-05-05 13:43:56,053 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /10.15.19.76 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
2022-05-05 13:43:56,323 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-xx.xxx.xx4.xx7:1270-43 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/xx.xxx.xx4.xx7:1270], selector=sun.nio.ch.EPollSelectorImpl#323eac4e, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost 'xx.xxx.xx4.xx7' peerPort 1270 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-11]
2022-05-05 13:43:56,323 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /xx.xxx.xx4.xx7 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-11]
2022-05-05 13:43:56,335 INFO [GroupCoordinator 0]: Dynamic member with unknown member id joins group console-consumer-27420 in Empty state. Created a new member id console-consumer-f9cb320c-a5cf-4e40-a686-6dbd1d37cdd6 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-3]
2022-05-05 13:43:56,340 INFO [GroupCoordinator 0]: Preparing to rebalance group console-consumer-27420 in state PreparingRebalance with old generation 0 (__consumer_offsets-26) (reason: Adding new member console-consumer-f9cb320c-a5cf-4e40-a686-6dbd1d37cdd6 with group instance id None) (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-6]
2022-05-05 13:43:59,342 INFO [GroupCoordinator 0]: Stabilized group console-consumer-27420 generation 1 (__consumer_offsets-26) with 1 members (kafka.coordinator.group.GroupCoordinator) [executor-Rebalance]
2022-05-05 13:43:59,359 INFO [GroupCoordinator 0]: Assignment received from leader console-consumer-f9cb320c-a5cf-4e40-a686-6dbd1d37cdd6 for group console-consumer-27420 for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-7]
2022-05-05 13:43:59,468 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-xx.xxx.xx4.xx7:1584-43 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/xx.xxx.xx4.xx7:1584], selector=sun.nio.ch.EPollSelectorImpl#5c2628ee, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost 'xx.xxx.xx4.xx7' peerPort 1584 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
2022-05-05 13:43:59,468 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /xx.xxx.xx4.xx7 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
Case 2 (which does not work, and is why I am asking this question)
Testing from my local machine (connecting through Internet).
Producer
$ /opt/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --bootstrap-server internet-reachable-domain-name:9094 --topic public.kafka.incoming.events --producer.config consumer-ssl.properties
....long SSL/TLS hand-shake....
)
ttesttttt
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-05-05 10:05:04.469 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
Notice how the character '>' is not getting displayed on the first line.
Consumer
$ /opt/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server internet-reachable-domain-name:9094 --topic public.kafka.incoming.events --consumer.config consumer-ssl.properties
javax.net.ssl|DEBUG|01|main|2022-05-05 10:08:29.034 EDT|Finished.java:535|Consuming server Finished handshake message (
"Finished": {
"verify data": {
0000: 6A 3A 3D F6 87 81 A0 36 FF 08 77 CB
}'}
)
javax.net.ssl|ALL|01|main|2022-05-05 10:08:37.815 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:08:48.159 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:09:06.713 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:09:37.193 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:10:11.936 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
Note how the message is not reaching, and even if I type a message from a Producer of the Case #1, its not showing on this consumer.
Confirmation
Confirmation based on the Kafka server logs.
When a Producer is connecting
2022-05-05 14:14:43,507 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-10.15.30.64:19952-46 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/10.15.30.64:19952], selector=sun.nio.ch.EPollSelectorImpl#5c2628ee, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost '10.15.30.64' peerPort 19952 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
2022-05-05 14:14:43,507 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /10.15.30.64 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
When a Consumer is connecting
2022-05-05 14:16:02,620 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-10.15.219.64:42153-47 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/10.15.219.64:42153], selector=sun.nio.ch.EPollSelectorImpl#7df0ddcc, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost '10.15.219.64' peerPort 42153 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
2022-05-05 14:16:02,621 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /10.15.219.64 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
Other information
I am using the option:
export KAFKA_OPTS="-Djavax.net.debug=ssl:handshake:verbose"
to get more information from Kafka clients.
This is what I can see from the producer in case #2.
javax.net.ssl|ALL|01|main|2022-05-05 09:54:50.374 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:55:23.285 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:55:48.477 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:56:13.718 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:56:50.532 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:57:25.793 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:58:01.707 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:58:30.806 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:59:07.867 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
Kafka version
Both clients are identical:
# Local device
$ /opt/kafka_2.13-3.1.0/bin/kafka-topics.sh --version
3.1.0 (Commit:37edeed0777bacb3)
# Kafka server pod
$ /opt/kafka/bin/kafka-topics.sh --version
3.1.0 (Commit:37edeed0777bacb3)
Thanks #OneCricketeer for the information.
The links you provided were the exact one I had to look. The problem was that Ingress option or Load-Balancer did not work on my Kubernetes Cluster, Kafka was not starting. Only nodePort option worked, but that was not sufficient to address the problem I had.
I ended up using bitnami instead of strimzi, https://bitnami.com/stack/kafka/helm and it worked like charm.
I relied on the example provided by #Rubber Duck on Accessing bitnami/kafka outside the kubernetes cluster
I just had to make sure to forward the ports by the load-balancer.
command:
helm install kafka-bitnami --values bitnami-values-external.yaml bitnami/kafka
file: bitnami-values-external.yaml
replicaCount: 3
externalAccess:
enabled: true
autoDiscovery:
enabled: false
image:
registry: docker.io
repository: bitnami/kubectl
tag: 1.23.4-debian-10-r17
pullPolicy: IfNotPresent
pullSecrets: []
resources:
limits: {}
requests: {}
service:
type: NodePort
port: 9094
loadBalancerIPs: []
loadBalancerSourceRanges: []
nodePorts:
- 30000
- 30001
- 30002
useHostIPs: false
annotations: {}
domain: <domain-name which points to my load-balancer public IP>
Note that replicaCount must much the number of nodePorts items.
load balancer configs
The load balancer config will have something like:
frontend-1 port: 30000 -> backend-1 port: 30000, server ips: [set of private worker node ips]
frontend-1 port: 30001 -> backend-1 port: 30001, server ips: [set of private worker node ips]
frontend-1 port: 30002 -> backend-1 port: 30002, server ips: [set of private worker node ips]
Maybe from here, I could come back to strimzi and try to adjust/fix it because I understand little bit better how it works.
Well, Strimzi is a great platform to run Kafka on Kubernetes. As you haven't shared the kafka CR file, I am assuming you have done the following setup to access Kafka from outside the k8s cluster-
# ...
listeners:
# ...
- name: external
port: 9094
type: loadbalancer
tls: true
# ...
Make sure you have resources like the below-
With this configuration and proper Authn/Authz, you would be able to access Kafka from outside the k8s cluster.
Check this article (PS - written by me), which takes a dig on how to enable Authn/Authz in Strimzi and it also discusses Accessing Kafka from inside the k8s cluster and from outside of it.
I am trying to setup inter-broker SSL (not client) authentication and keep seeing the following errors:
[2019-05-17 06:33:47,151] INFO [Controller id=1004, targetBrokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] INFO [SocketServer brokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] ERROR [Controller id=1004, targetBrokerId=1004] Connection to node 1004 (/$IP:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
My server.properties is:
listeners=PLAINTEXT://$IP:9092,SSL://$IP:9093
security.inter.broker.protocol=SSL
ssl.truststore.password=$PASS
ssl.keystore.password=$PASS
ssl.key.password=$PASS
ssl.endpoint.identification.algorithm=""
ssl.keystore.location=/etc/kafka/kafka.server.keystore.jks
ssl.truststore.location=/etc/kafka/kafka.server.truststore.jks
``
When I run `openssl s_client -debug -connect $IP:9093 -tls1` I get back a list of certificates and `Secure Renegotiation IS supported`
Despite adding `-Djavax.net.debug=all` there's not anything in the logs which points to the problem.
Kafka version 2.2
Any ideas?
I had incorrectly set the value of ssl.endpoint.identification.algorithm="" instead of ssl.endpoint.identification.algorithm", this fixed it.
This value was changed in 2.2 to default to https so setting it to nothing worked.
I have made a simple RESTlet that I have hosted on an Amazon EC2 instance.
I can reach this when i run it over HTTP.
I needed to convert this to working with HTTPS so followed the RESTlet guide at https://restlet.com/open-source/documentation/user-guide/2.2/core/security/https
When I connect to it over HTTP I get this error..
Restlet-917142466, READ: TLSv1.2 Alert, length = 2
Restlet-917142466, RECV TLSv1.2 ALERT: fatal, certificate_unknown
Restlet-917142466, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
Restlet-917142466, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
Restlet-917142466, called closeInbound()
Restlet-917142466, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
Restlet-917142466, called closeOutbound()
Restlet-917142466, closeOutboundInternal()
The Java code is very simple and mirrors the sample RESTlet server in the guide.
Any suggestions on what I could try to fix this?
I write a demo to access a server with java HttpsUrlConnection.
I implement the X509TrustManager interface with MyX509TrustManager to trust all certificates.
It works fine with JDK7 and JDK8, but when use JDK6 and JDK5, error happens:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1014)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
For server side, running on Tomcat 8 and JDK8.
EDIT I add -Djavax.net.debug=ssl,handshake and find following:
*** main, WRITE: TLSv1 Handshake, length = 177 main, WRITE: SSLv2 client hello message, length = 173
main, received EOFException: error main, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
main, SEND TLSv1 ALERT: fatal, description = handshake_failure
main, WRITE: TLSv1 Alert, length = 2 main, called closeSocket()
main, called close() main, called closeInternal(true)