I'm testing kafka cluster creation using let's encrypt staging certs. After creating, on my machine, I run the kafka-provided kafka-console-consumer.sh and kafka-console-producer.sh scripts. When I ran with let's encrypt production, it worked fine. But now that I'm using staging certs, I get this when I run the producer:
ERROR [Producer clientId=console-producer] Connection to node -1 (2.kafka.mysite.com/10.1.17.191:9092) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
I use these properties for producer script:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="secret";
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
I'd like to give the option to ignore TLS, and I'd like it to be some parameter I can toggle (on the cluster or on the client) to allow it. How can I achieve this? For anyone familiar with Rabbitmq, I think it's similar to VERIFY_PEER=false, aka VERIFY_NONE.
The kafka configuration has setting
ssl.client.auth
Its value could be set as required,requested or none. You could set it to requested.his means client authentication is optional. unlike requested , if this option is set client can choose not to provide authentication information about itself
https://docs.confluent.io/current/installation/configuration/broker-configs.html
Related
I'm trying to enable SSL on a Artemis broker and always get this exception when trying to connect:
Exception in thread "main" ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:743)
The code I use to connect is just this:
ActiveMQClient.createServerLocator("tcp://localhost:5500").createSessionFactory();
This is from a fresh install of Artemis 2.23.1 and the only thing I changed from the default broker configuration was to add this acceptor in broker.xml:
<acceptor name="netty-ssl-acceptor">tcp://localhost:5500?sslEnabled=true;keyStorePath=server-keystore.jks;keyStorePassword=securepass</acceptor>
I generated the keystore and truststore using the script provided in this example.
I had first tried a keystore with a cert that is valid for my domain (using a domain-qualified host name in createServerLocator()) but that also gave me the timeout. That is when I went back to fresh installs and tried going through the SSL example.
Various attempts with invalid paths/passwords/certs threw exceptions that led me to what to fix, but so far haven't been able to see what I did wrong with a generic timeout discovering cluster topology.
Anybody have ideas?
You need to specify sslEnabled=true on the client's URL as well so it knows to use SSL, e.g.:
ActiveMQClient.createServerLocator("tcp://localhost:5500?sslEnabled=true").createSessionFactory();
This is done for the JMS connection in the ssl-enabled example which you cited here.
Also, if you're using self-signed certificates then you'll need a truststore for your client as well and you'll need to configure those settings on the client's URL (just like in the example).
I am new to Apache Kafka, and here is what I have done so far,
Downloaded kafka_2.12-2.1.0
Make Batch file for Zookeeper to run zookeeper server:
start kafka_2.12-2.1.0.\bin\windows\zookeeper-server-start.bat kafka_2.12-2.1.0.\config\zookeeper.properties
Make Batch File for Apache Kafka server
start kafka_2.12-2.1.0\bin\windows\kafka-server-start.bat kafka_2.12-2.1.0\config\server.properties
Started A Producer using batch file.
start kafka_2.12-2.1.0.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic 3drocket-player
It is running fine but now I am looking for authentication. As I have to implement a consumer with specific auth settings (requirement by the client). Like security protocol is SASL_SSL and SSL mechanism is GSSAPI.
For this reason, I tried to search and find confluet documentation but the problem is it is too abstract that how to take each and every step.
I am looking for detail configuration steps according to my setup. How to configure my kafka server with SASL SSL and GSSAPI protocol. Initially I found that GSSAPI/Keberos has a separate server then, do i need to install more server? Within Confluent Kafka is there any built-in solution.
Configure a SASL port in server.properties
e.g)
listeners=SASL_SSL://host.name:port
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=keystore_password
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=truststore_password
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
https://kafka.apache.org/documentation/#security_configbroker
https://kafka.apache.org/documentation/#security_sasl_config
Client:
When you run the Kafka client, you need to set these properties.
security.protocol=SASL_SSL
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=truststore_password
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
https://kafka.apache.org/documentation/#security_configclients
https://kafka.apache.org/documentation/#security_sasl_kerberos_clientconfig
Then configure the JAAS configuration
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="path/to/kafka_client.keytab"
storeKey=true
useTicketCache=false
principal="kafka-client-1#EXAMPLE.COM";
};
...
SASL/GSSAPI is for organizations using Kerberos (for example, by using Active Directory). You don’t need to install a new server just for Apache Kafka®. Ask your Kerberos administrator for a principal for each Kafka broker in your cluster and for every operating system user that will access Kafka with Kerberos authentication (via clients and tools).
https://docs.confluent.io/current/kafka/authentication_sasl/authentication_sasl_gssapi.html#kafka-sasl-auth-gssapi
....
I am using kafka version 2.12-2.2.1 in windows operating system. I have implemented tls on my local windows systems using the process of signed certificates.
The kafka is running fine and there is one command to check if certificates are installed on kafka.
openssl s_client -debug -connect localhost:9093 -tls1
But when I try to connect to localhost:9093 and using producer or consumer it is throwing me an error saying :
connection to node -1 failed due to authentication
I have tried everything, I am stuck even the documentation provided is not giving any hints to solve this error.
Note: One more addition how can I see the list of topics and describe the topic if exits using ssl in kafka because that command is also not working.
Along with that I have tried every answer on SO but still no success.
The documentation I have followed Installing ssl on kafka
I am having issue connecting to a qmgr. the host rejected connection due to cipherspec error for ssl channel on port 1414. The keystore checked out ok. I was able to use openssh to connect to the host and retrieve its keys.
I have tried to enable and disable sslv3. I provided keystore password with and without "" (double quotes). These are connection properties
qcf=wmq://aftbusu105.it.companyx.com:1414/?qmgr=MQPLTC010,channel=FUSION.SSL,sslCipherSuite=SSL_RSA_WITH_NULL_MD5,transportType=1
reqQ=queue:///FUSIONQL.app.queuename.1_0.Q.PS.REQ
rspQ=queue:///FUSIONQL.app.queuename.1_0.Q.PS.REQ
mep=oneway
connCnt=1
sessCnt=1
numMsgs=1
connInterval=10
msgInterval=10
deliveryMode=1
priority=1
expiration=1
keystore=/path/keystore/m36797q.jks
password=a$tilBe2Flower
alias=m36797q
Do you know what the issue could be?
Can you confirm if you have FIPs enabled on either the server or the client? It's possible you are getting error because the ciphersuite
SSL_RSA_WITH_NULL_MD5 is not supported in FIPS mode. Are you seeing any AMQ errors in your QMGR error logs?
Also, let us know the MQ version you are using.
I'm using django-celery do connect to a RabbitMQ broker through SSL (with the BROKER_USE_SSL setting). Is there a way to:
Verify the certificate of the broker when the connection is established.
Configure a client certificate to us to establish the connection.
The RabbitMQ side is working correctly, but I don't know how to configure Celery for this and I haven't found anything in Celery's documentation either. The settings CELERY_SECURITY_KEY, CELERY_SECURITY_CERTIFICATE and CELERY_SECURITY_CERT_STORE look like they could do this, but it seems that they're only used for message signing.
kombu.Connection accepts ssl argument as a dictionary of SSL configuration (ssl=False by default). I suppose it is applicable for BROKER_USE_SSL too.
BROKER_USE_SSL={
'ca_certs': '/etc/pki/tls/certs/something.crt',
'keyfile': '/etc/something/system.key',
'certfile': '/etc/something/system.cert',
'cert_reqs': ssl.CERT_REQUIRED,
}