Not Able to Update Replication Factor of Kafka Topic - ssl

Question

We have a 3 node kafka-zookeeper cluster setup with kafka-zookeeper communicating on SSL.
We are currently using apache kafka 2.5 and zookeeper 3.5.7 . We are trying to increase the replication factor in kafka topics using the below method:
To increase the number of replicas for a given topic you have to:
Specify the extra replicas in a custom reassignment json file
For example, you could create increase-replication-factor.json and put this content in it:
{"version":1,
"partitions":[
{"topic":"signals","partition":0,"replicas":[0,1,2]},
{"topic":"signals","partition":1,"replicas":[0,1,2]},
{"topic":"signals","partition":2,"replicas":[0,1,2]}
]}
Use the file with the --execute option of the kafka-reassign-partitions tool
[or kafka-reassign-partitions.sh - depending on the kafka package]
For example:
$ kafka-reassign-partitions --zookeeper localhost:2182 --reassignment-json-file increase-replication-factor.json --execute --command-config zookeeper_client.properties
But we are facing the problem while running the kafka-reassign-partitions , while running this command the connection to zookeeper fails with below error:
2022-04-28 05:56:46,963 [myid:1] - ERROR [nioEventLoopGroup-7-3:NettyServerCnxnFactory$CertificateVerifier#363]
Unsuccessful handshake with session 0x0
2022-04-28 05:56:46,963 [myid:1] - WARN [nioEventLoopGroup-7-3:NettyServerCnxnFactory$CnxnChannelHandler#220]
Exception caught
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1198)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
We are passing all the certificate and keystore data through --command-config , the zookeeper_client.properties is as below:
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2
zookeeper.ssl.truststore.location=kafka.truststore.jks
zookeeper.ssl.truststore.password=changeme
zookeeper.ssl.keystore.location=kafka.keystore.jks
zookeeper.ssl.keystore.password=changeme
zookeeper.ssl.endpoint.identification.algorithm=
zookeeper.ssl.hostnameVerification=false
We have also tried to set CLIENT_JVMFLAGS and KAFKA_OPTS with same jvm arguments but that doesn't help.
The option of passing zookeeper_client.properties via -zk-tls-config-file is not available in kafka-reassign-partitions.sh.