I am trying to establish 2 way TLS authentication with Kafka (version 2.4.0).
My server.properties configuration:
listeners=SSL://kafka1:9092
advertised.listeners=SSL://<public_ip>:9092
ssl.keystore.location=path/server.keystore.jks
ssl.keystore.password=pass
ssl.truststore.location=path/server.truststore.jks
ssl.truststore.password=pass
ssl.endpoint.identification.algorithm=HTTPS
security.inter.broker.protocol=SSL
client.properties:
bootstrap.servers=kafka1:9092
security.protocol=SSL
ssl.keystore.location=path/client.keystore.jks
ssl.keystore.password=pass
ssl.truststore.location = path/client.truststore.jks
ssl.truststore.password=pass
kafkaServer and zooKeeper are on.
But when I try to create topic:
kafka-topics.sh --bootstrap-server kafka1:9092 --create --replication-factor 1 --partitions 1 --topic sometopic --command-config config/client.properties
get an error:
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No name matching kafka1 found
My certificate connected to kafka1 name, and there is mapping of kafka1 to local_ip.
Have you seen such an error before?
Related
I have a Apache Kafka (v. 2.13-3.0.0) installed on a remote Ubuntu server.
I follow this tutorial to secure my cluster:
https://medium.com/egen/securing-kafka-cluster-using-sasl-acl-and-ssl-dec15b439f9d
but when I try to start Kafka with jaas conf file with the commands:
export KAFKA_OPTS=-Djava.security.auth.login.config=<kafka-binary-
dir>/config/kafka_server_jaas.conf
./bin/kafka-server-start.sh ./config/server.properties
I receive the error:
[2021-11-12 10:30:47,864] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-11-12 10:30:48,089] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-11-12 10:30:48,099] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.ClassNotFoundException: kafka.security.auth.SimpleAclAuthorizer
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.kafka.common.utils.Utils.loadClass(Utils.java:417)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:406)
at kafka.security.authorizer.AuthorizerUtils$.createAuthorizer(AuthorizerUtils.scala:31)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1583)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1394)
at kafka.Kafka$.buildServer(Kafka.scala:67)
at kafka.Kafka$.main(Kafka.scala:87)
at kafka.Kafka.main(Kafka.scala)
These are the SSL config in server.properties file:
########### SECURITY using SCRAM-SHA-512 and SSL
listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://localhost:9093,SASL_SSL://localhost:9094
advertised.listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://localhost:9093,SASL_SSL://localhost:9094
security.inter.broker.protocol=SASL_SSL
ssl.endpoint.identification.algorithm=
ssl.client.auth=required
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
sasl.enabled.mechanisms=SCRAM-SHA-512
# Broker security settings
ssl.truststore.location=/home/kafka/Downloads/kafka_2.13-3.0.0/config/truststore/kafka.truststore.jks
ssl.truststore.password=giuseppe
ssl.keystore.location=/home/kafka/Downloads/kafka_2.13-3.0.0/config/keystore/kafka.keystore.jks
ssl.keystore.password=giuseppe
ssl.key.password=giuseppe
# ACLs
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin
#zookeeper SASL
zookeeper.set.acl=false
########### SECURITY using SCRAM-SHA-512 and SSL
If I try to comment the 2 rows of ACL I receive the error:
[2021-11-12 11:05:29,301] INFO [ThrottledChannelReaper-
ControllerMutation]: Starting
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-11-12 11:05:29,331] ERROR [KafkaServer id=0] Fatal error
during KafkaServer startup. Prepare to shutdown
(kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Failed to acquire lock on
file .lock in /tmp/kafka-logs. A Kafka instance in another process
or thread is using this directory.
at kafka.log.LogManager.$anonfun$lockLogDirs$1(LogManager.scala:241)
at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:117)
at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:104)
at scala.collection.mutable.ArraySeq.flatMap(ArraySeq.scala:37)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:236)
at kafka.log.LogManager.<init>(LogManager.scala:112)
at kafka.log.LogManager$.apply(LogManager.scala:1283)
at kafka.server.KafkaServer.startup(KafkaServer.scala:254)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
What is the cause? May it be a wrong configuration?
Thanks.
Update:
Changing the row in:
# ACLs authorizer.class.name=org.apache.kafka.server.authorizer.Authorizer
there is this error: org.apache.kafka.common.KafkaException: Could not find
a public no-argument constructor for
org.apache.kafka.server.authorizer.Authorizer at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:392)
I receive this new error:
[2021-11-12 16:51:57,613] ERROR Exiting Kafka due to fatal exception
(kafka.Kafka$)
org.apache.kafka.common.KafkaException: Could not find a public no-argument
constructor for org.apache.kafka.server.authorizer.Authorizer at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:392)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:406)
at kafka.security.authorizer.AuthorizerUtils$.createAuthorizer(AuthorizerUtils.scala:31)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1583)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1394)
at kafka.Kafka$.buildServer(Kafka.scala:67)
at kafka.Kafka$.main(Kafka.scala:87)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.NoSuchMethodException:
org.apache.kafka.server.authorizer.Authorizer.<init>()
at java.base/java.lang.Class.getConstructor0(Class.java:3508)
at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2711)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:390)
... 7 more
It just seems that if you change the
kafka.security.auth.SimpleAclAuthorizer
to
kafka.security.authorizer.AclAuthorizer
It should work; it worked for me.
Kafka 3.0 removed SimpleAclAuthorizer
Pull request - https://github.com/apache/kafka/commit/976e78e405d57943b989ac487b7f49119b0f4af4#diff-e0ccf1b5c964d2c303b6a69a8b8b67df5a6bfbae8aa514f580d353c4c6bf8e36
The blog seems to be using version 2.2.0.
I'm trying to publish message to a Tibco Queue on a SSL Tibco Server through JMeter 5.4.1 using JMS Point-to-Point Logic Controller.
JMS Point To Point Controller Config
But I'm getting the following error message:
2021-06-13 12:25:46,278 ERROR o.a.j.p.j.s.JMSSampler: Not permitted:
Failed to connect to any server at: ssl://[server-name]:7352,
ssl://[server-name]:7352 [Error: Failed to connect via SSL to
[ssl://[server-name]:7352]: Received fatal alert:
protocol_version: url that returned this exception =
SSL://[server-name]:7352 ]
javax.naming.AuthenticationException: Not permitted: Failed to connect
to any server at: ssl://[server-name]:7352,
ssl://[server-name]:7352 [Error: Failed to connect via SSL to
[ssl://[server-name]:7352]: Received fatal alert:
protocol_version: url that returned this exception =
SSL://[server-name] ] at
com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:670)
~[tibjms.jar:8.0.0] at
com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:491)
~[tibjms.jar:8.0.0] at
javax.naming.InitialContext.lookup(InitialContext.java:417)
~[?:1.8.0_291] at
org.apache.jmeter.protocol.jms.sampler.JMSSampler.threadStarted(JMSSampler.java:638)
[ApacheJMeter_jms.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:784)
[ApacheJMeter_core.jar:5.4.1] at
org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:993)
[jorphan.jar:5.4.1] at
org.apache.jorphan.collections.HashTree.traverse(HashTree.java:976)
[jorphan.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread.threadStarted(JMeterThread.java:752)
[ApacheJMeter_core.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread.initRun(JMeterThread.java:740)
[ApacheJMeter_core.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:252)
[ApacheJMeter_core.jar:5.4.1]
I tried:
openssl s_client -connect [server-name]:7352
It gave the following output:
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID:
Session-ID-ctx:
So added the following line in jmeter.properties file.
https.default.protocol=TLSv1.2
Also commented jdk.tls.disabledAlgorithms from java.security file for JDK (I'm using jdk1.8.0_291)
# jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
# DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
# include jdk.disabled.namedCurves
But still I'm getting the same error. Someone please help.
I think you're using the wrong property (not only the wrong property but the wrong place as well), you're setting default protocol for HTTPS, while you need to set it for TLS, i.e. add the next line to system.properties file
jdk.tls.client.protocols=TLSv1.2
JMeter restart will be required to apply this property.
If it won't help or you will get different errors - consider adding the next line there as well:
javax.net.debug=all
and then check jmeter.log file and stdout for any suspicious entries
More information:
Configuring JMeter
Apache JMeter Properties Customization Guide
I resolved it by using the latest tibjms.jar in the lib directory in JMeter as the Tibco server was upgraded some hours before I raised this issue.
I've configured both Kafka server and Zookeeper server to use SSL/TLS using the JKS. I've confirmed this using openssl. I'm using Bitnami Helm charts of Kafka and Zookeeper. Below is the log output from Kafka. I'm pretty sure that the Kafka client isn't sending requests to Zookeeper server securely because of the Zookeeper logs. How do I ensure Kafka client uses SSL/TLS. I think the kafka client needs to use a client.properties file when executing config commands with args. But I don't know how to pass this file in during configuration. The logs show that Kafka client is trying to add a user called zookeeperUser to Zookeeper. This communication is non secure.
Kafka Logs
09:56:31.43
09:56:31.43 Welcome to the Bitnami kafka container
09:56:31.44 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
09:56:31.44 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
09:56:31.44
09:56:31.44 INFO ==> ** Starting Kafka setup **
09:56:31.56 DEBUG ==> Validating settings in KAFKA_* env vars...
09:56:31.65 INFO ==> Initializing Kafka...
09:56:31.66 INFO ==> No injected configuration files found, creating default config files
09:56:32.96 INFO ==> Configuring Kafka for inter-broker communications with SASL_SSL authentication.
09:56:33.13 INFO ==> Configuring Kafka for client communications with SASL_SSL authentication.
09:56:33.43 INFO ==> Custom JAAS authentication file detected. Skipping generation.
09:56:33.43 WARN ==> The following environment variables will be ignored: KAFKA_CLIENT_USERS, KAFKA_CLIENT_PASSWORDS, KAFKA_INTER_BROKER_USER, KAFKA_INTER_BROKER_PASSWORD, KAFKA_ZOOKEEPER_USER and KAFKA_ZOOKEEPER_PASSWORD
09:56:33.44 INFO ==> Creating users in Zookeeper
09:56:33.44 DEBUG ==> Creating user zookeeperUser in zookeeper
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Error while executing config command with args '--zookeeper zookeeper.default.svc.cluster.local:3181 --alter --add-config SCRAM-SHA-256=[iterations=8192,password=zookeeperPassword],SCRAM-SHA-512=[password=zookeeperPassword] --entity-type users --entity-name zookeeperUser'
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1881)
at kafka.admin.ConfigCommand$.processCommandWithZk(ConfigCommand.scala:116)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:94)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
client.properties
cat > client.properties <<EOF
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
ssl.truststore.location=/tmp/kafka.truststore.jks
ssl.truststore.password=******
EOF
Zookeeper Logs
2021-02-11 09:56:43,055 [myid:1] - ERROR [nioEventLoopGroup-7-1:NettyServerCnxnFactory$CertificateVerifier#434] - Unsuccessful handshake with session 0x0
2021-02-11 09:56:43,055 [myid:1] - WARN [nioEventLoopGroup-7-1:NettyServerCnxnFactory$CnxnChannelHandler#273] - Exception caught
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
... 17 more
I have a distributed kafka with 3 brokers that have port numbers 9093, 9094, 9095 and I added the SSL with port numbers 9096, 9097, 9098.
I am getting the following error when I run the producer client:
[2020-06-15 10:08:07,892] ERROR [Producer clientId=console-producer] Connection to node -1 (/myip-address:9096) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-06-15 10:08:07,893] WARN [Producer clientId=console-producer] Bootstrap broker myip-address:9096 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
I have tried to use: ssl.endpoint.identification.algorithm= but that did not solve the problem for me.
I am using Kafka 2.5
I can share my config files if need be.
What else could I try to troubleshoot this issue?
Thank you.
I was able to solve this issue by simply using my
domain-name:9096,domain-name:9097,domain-name:9098
instead of:
my-ip-address:9096,my-ip-address:9097,my-ip-address:9098
So, using the actual domain name is important due to the certificate being created with the domain name.
I am trying to setup inter-broker SSL (not client) authentication and keep seeing the following errors:
[2019-05-17 06:33:47,151] INFO [Controller id=1004, targetBrokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] INFO [SocketServer brokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] ERROR [Controller id=1004, targetBrokerId=1004] Connection to node 1004 (/$IP:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
My server.properties is:
listeners=PLAINTEXT://$IP:9092,SSL://$IP:9093
security.inter.broker.protocol=SSL
ssl.truststore.password=$PASS
ssl.keystore.password=$PASS
ssl.key.password=$PASS
ssl.endpoint.identification.algorithm=""
ssl.keystore.location=/etc/kafka/kafka.server.keystore.jks
ssl.truststore.location=/etc/kafka/kafka.server.truststore.jks
``
When I run `openssl s_client -debug -connect $IP:9093 -tls1` I get back a list of certificates and `Secure Renegotiation IS supported`
Despite adding `-Djavax.net.debug=all` there's not anything in the logs which points to the problem.
Kafka version 2.2
Any ideas?
I had incorrectly set the value of ssl.endpoint.identification.algorithm="" instead of ssl.endpoint.identification.algorithm", this fixed it.
This value was changed in 2.2 to default to https so setting it to nothing worked.