ActiveMQ mutual SSL authentication - ssl

I'm trying to set up ActiveMQ for mutual authentication, that the client will need a certificate in order to pass messages to the broker. I created a keystore and a truststore on the broker and exported a certificate that was copied to the client. On the client side I did the same, although I am using NMS so I am only using the exported certificate, which I added to the broker's truststore. I also added the certificate's to the other's Local Computer Trusted Root Certificates.
The broker's configuration has this:
<transportConnectors>
<transportConnector name="ssl" uri="ssl://0.0.0.0:61616"/>
</transportConnectors>
<sslContext>
<sslContext keyStore="file:${activemq.base}/conf/keystore.jks"
keyStorePassword="ksPass"
trustStore="file:${activemq.base}/conf/shared.ks"
trustStorePassword="ksPass"/>
</sslContext>
<plugins>
<jaasCertificateAuthenticationPlugin configuration="CertLogin" />
</plugins>
amq service wrapper.conf has
wrapper.java.additional.8=-Djava.security.auth.login.config="%ACTIVEMQ_CONF%/login.config"
${activemq.base}/conf/login.config
CertLogin {
org.apache.activemq.jaas.TextFileCertificateLoginModule required
debug=true
org.apache.activemq.jaas.textfiledn.user="users.properties"
org.apache.activemq.jaas.textfiledn.group="groups.properties";
};
${activemq.base}/conf/users.properties has
user=CN=nms.client.170,\ OU=IT,\ O=MyOrg,\ L=Oslo,\ S=Oslo,\ C=NO
${activemq.base}/conf/groups.properties has
admins=system
users=system,user
In the NMS client's appSettings I use this to connect:
< add key="jms.uri" value="ssl://brokeraddress.in.hosts:61616?needClientAuth=true&wantClientAuth=true&transport.clientCertSubject=nms.client.170&transport.clientCertPassword=ksClientPw&transport.clientCertFilename=C:\TestClient\client170.crt" />
If I don't have the jaasCertificateAuthenticationPlugin in the broker I can connect via ssl, but with it (which is what I had in mind) I get an error, it fails in Apache.NMS.ActiveMQ.Connection
// Send the connection and see if an ack/nak is returned.
Response response = transport.Request(this.info, this.RequestTimeout);
Gets an ExeptionResponse: "java.lang.SecurityException: Unable to authenticate transport without SSL certificate."
The trace shows:
10:19:16,479 INFO Client.MyTrace - BrokerUri set = ssl://brokeraddress.in.hosts:61616?transport.clientcertpassword=ksClient&transport.clientcertsubject=nms.client.170&needclientauth=true&wantclientauth=true&transport.clientcertfilename=C:\TestClient\client170.crt
10:19:16,492 DEBUG Client.MyTrace - SetProperties called with target: ConnectionFactory, and prefix: connection.
10:19:16,492 DEBUG Client.MyTrace - SetProperties called with target: ConnectionFactory, and prefix: nms.
10:19:16,495 INFO Client.MyTrace - Connecting to: ssl://brokeraddress.in.hosts:61616/?transport.clientcertpassword=ksClient&transport.clientcertsubject=nms.client.170&needclientauth=true&wantclientauth=true&transport.clientcertfilename=C:\TestClient\client170.crt
10:19:16,497 DEBUG Client.MyTrace - Searching Assembly: Apache.NMS.ActiveMQ for factory of the id: ssl
10:19:16,549 DEBUG Client.MyTrace - Found the Factory of type Apache.NMS.ActiveMQ.Transport.Tcp.SslTransportFactory for id: ssl
10:19:16,552 DEBUG Client.MyTrace - Opening socket to: brokeraddress.in.hosts on port: 61616
10:19:16,554 DEBUG Client.MyTrace - Connected to brokeraddress.in.hosts:61616 using InterNetwork protocol.
10:19:16,562 DEBUG Client.MyTrace - Creating new instance of the SSL Transport.
10:19:16,564 DEBUG Client.MyTrace - Creating Inactivity Monitor: 1
10:19:16,677 DEBUG Client.MyTrace - Authorizing as Client for Server: brokeraddress.in.hosts
10:19:16,679 DEBUG Client.MyTrace - Attempting to load Client Certificate from file := C:\TestClient\client170.crt
10:19:16,682 DEBUG Client.MyTrace - Loaded Client Certificate := [Subject] CN=nms.client.170, OU=IT, O=MyOrg, L=Oslo, S=Oslo, C=NO [Issuer] CN=nms.client.170, OU=IT, O=MyOrg, L=Oslo, S=Oslo, C=NO
10:19:16,684 DEBUG Client.MyTrace - Client is selecting a local certificate from 1 possibilities.
10:19:16,684 DEBUG Client.MyTrace - Client has selected certificate with Subject = CN=nms.client.170, OU=IT, O=MyOrg, L=Oslo, S=Oslo, C=NO
10:19:16,969 DEBUG Client.MyTrace - ValidateServerCertificate: Issued By CN=brokeraddress.in.hosts, OU=DataCom, O=MyOrg, L=Oslo, S=Oslo, C=NO
10:19:16,969 DEBUG Client.MyTrace - Server is Authenticated = True
10:19:16,970 DEBUG Client.MyTrace - Server is Encrypted = True
10:19:16,978 DEBUG Client.MyTrace - InactivityMonitor[1]: Read Check time interval: 30000
10:19:16,978 DEBUG Client.MyTrace - InactivityMonitor[1]: Initial Delay time interval: 10000
10:19:16,985 DEBUG Client.MyTrace - InactivityMonitor[1]: Write Check time interval: 10000
10:19:19,017 DEBUG Client.MyTrace - Exception received in the Inactivity Monitor: Unable to read beyond the end of the stream.
10:19:19,019 DEBUG Client.MyTrace - InactivityMonitor[1].Runner: Task Runner Shut Down
10:19:19,019 DEBUG Client.MyTrace - InactivityMonitor[1]: Stopped Monitor Threads.
10:19:19,032 DEBUG Client.MyTrace - Connection[ID:EJPB-56409-635193299565662525-1:0]: Async exception with no exception listener: System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
System.IO.BinaryRe.FillBuffer(Int32 numBytes)
System.IO.BinaryRe.ReadInt32()
Apache.NMS.Util.EnBinaryReader.ReadInt32() in c:\dev\NMS\src\main\csharp\Util\EndianBinaryReader.cs:line 135
Apache.NMS.ActiveMenWire.OpenWireFormat.Unmarshal(BinaryReader dis) in c:\dev\NMS.ActiveMQ\src\main\csharp\OpenWire\OpenWireFormat.cs:line 228
Apache.NMS.ActiveMansport.Tcp.TcpTransport.ReadLoop() in c:\dev\NMS.ActiveMQ\src\main\csharp\Transport\Tcp\TcpTransport.cs:line 295
10:19:19,035 DEBUG Client.MyTrace - TransportFilter disposing of next Transport: MutexTransport
10:19:19,035 DEBUG Client.MyTrace - TransportFilter disposing of next Transport: WireFormatNegotiator
10:19:19,036 DEBUG Client.MyTrace - TransportFilter disposing of next Transport: InactivityMonitor
10:19:19,036 DEBUG Client.MyTrace - InactivityMonitor[1]: Stopped Monitor Threads.
10:19:19,037 DEBUG Client.MyTrace - TransportFilter disposing of next Transport: SslTransport
10:19:19,071 INFO Client.MyTrace - Connection[ID:SUSSDEV2-56409-635193299565662525-1:0]: Closing Connection Now.
10:19:19,073 DEBUG Client.MyTrace - Connection[ID:SUSSDEV2-56409-635193299565662525-1:0]: Disposing of the Transport.
10:19:19,073 DEBUG Client.MyTrace - InactivityMonitor[1]: Stopped Monitor Threads.
and in the broker it says:
INFO | jvm 1 | 10:18:20 | WARN | Failed to add Connection ID:EJPB-56409-635193299565662525-1:0, reason: java.lang.SecurityException: Unable to authenticate transport without SSL certificate.
INFO | jvm 1 | 10:18:22 | INFO | Stopping tcp://192.168.5.170:56408 because Failed with SecurityException: Unable to authenticate transport without SSL certificate.
Ok, what am I missing? It says "transport without SSL certificate", but it selects it during the client's connection and it is in both the broker's truststore and Root Certs.
Using NMS 1.6.0 and activeMQ 5.8.0.
I also tried a simple client in java with the same result.
Exception in thread "main" javax.jms.JMSException: Unable to authenticate transport without SSL certificate.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at org.apache.activemq.ActiveMQConnection.syncSendPacket (ActiveMQConnection.java:1295)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent (ActiveMQConnection.java:1392)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:504)
at com.atest.jms.Client.main(Client.java:69)
Caused by: java.lang.SecurityException: Unable to authenticate transport without SSL certificate.
at org.apache.activemq.security.JaasCertificateAuthenticationBroker.addConnection(JaasCertificateAuthenticationBroker.java:74)
at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:91)
at org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:766)
at org.apache.activemq.broker.jmx.ManagedTransportConnection.processAddConnection(ManagedTransportConnection.java:79)
at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Unknown Source)
Anyone has any ideas? Thanks.
Edit nov 12th
Ok, I tried again. I see I hadn't added to the broker's transportConnector ssl:
<transportConnector name="ssl" uri="ssl://0.0.0.0:61616?needClientAuth=true"/>
I tried the java client and it could send, but I'm still getting error for the NMS client (same config):
15:28:14,044 ERROR Test_DataCom.MyTrace - Exception: A call to SSPI failed, see inner exception.
15:28:14,045 ERROR Test_DataCom.MyTrace - Inner exception: An unknown error occurred while processing the certificate
15:28:14,045 ERROR Test_DataCom.MyTrace - Authentication failed - closing the connection.
and at the broker I get null cert chain
INFO | jvm 1 | 15:28:13 | ERROR | Could not accept connection from tcp://192.168.50.170:61978: javax.net.ssl.SSLHandshakeException: null cert chain
I have the certs in each other's truststore and Local Computer Trusted Root CA... what more is needed? Where should it find the cert chain for self signed certs if not there?

OK, I seem to have it now. After re-reading this comment I exported the client cert and key to a PKCS file and imported it to Current User Certification Store in Windows.
C:\> keytool -v -importkeystore -srckeystore client-keystore.jks -srcalias client -destkeystore client.p12 -deststoretype PKCS12
Then I used this url-encoded connection:
<add key="jms.uri" value="ssl://brokeraddress.in.hosts:61616?needClientAuth=true&wantClientAuth=true&transport.clientCertSubject=CN%3Dnms.client.170%2C+OU%3DIT%2C+O%3DMyOrg%2C+L%3DOslo%2C+S%3DOslo%2C+C%3DNO" />
I also had
${activemq.base}/conf/users.properties: user=CN=nms.client.170,\ OU=IT,\ O=MyOrg,\ L=Oslo,\ S=Oslo,\ C=NO, trying to escape the spaces, I removed those and corrected S to ST, just as keytool reports the Owner line. I checked, and that gives error otherwise.
Corrected ${activemq.base}/conf/users.properties:
user=CN=nms.client.170, OU=IT, O=MyOrg, L=Oslo, ST=Oslo, C=NO

Related

Datastax Driver giving connection error after enabling client to node SSL on Cassandra port 9142

Enabled SSL on cassandra nodes on port 9142. The service is running fine when testing it from local but getting AllNodesFailedException when deploying on ECS cluster. Using the same keystore locally. Non SSL Port 9042 is working ok.
Failed to instantiate [com.datastax.oss.driver.api.core.CqlSession]:
Factory method 'session' threw exception; nested exception is
com.datastax.oss.driver.api.core.AllNodesFailedException: Could not
reach any contact point, make sure you've provided valid addresses
(showing first 3 nodes, use getAllErrors() for more):
Node(endPoint=ip-10-18-28-203.us-west-2.compute.internal/10.18.28.203:9142,
hostId=null, hashCode=6551c917):
[io.netty.channel.ConnectTimeoutException: connection timed out:
ip-10-18-28-203.us-west-2.compute.internal/10.18.28.203:9142],
Node(endPoint=ip-10-18-8-110.us-west-2.compute.internal/10.18.8.110:9142,
hostId=null, hashCode=36985f57):
[io.netty.channel.ConnectTimeoutException: connection timed out:
ip-10-18-8-110.us-west-2.compute.internal/10.18.8.110:9142],
Node(endPoint=ip-10-18-7-47.us-west-2.compute.internal/10.18.7.47:9142,
hostId=null, hashCode=8eab7e9):
[io.netty.channel.ConnectTimeoutException: connection timed out:
ip-10-18-7-47.us-west-2.compute.internal/10.18.7.47:9142]
cassandra.yaml properties
server_encryption_options:
internode_encryption: none
keystore: /etc/cassandra/conf/casskeystore
keystore_password: changeit
truststore: conf/.truststore
truststore_password: cassandra
client_encryption_options:
enabled: true
optional: true
keystore: /etc/cassandra/conf/casskeystore
keystore_password: changeit

Securing NIFI with SSL Comodo Cert

I'm trying to secure our NIFI environment with SSL. I'm gettin the following error:
This site can’t provide a secure connection <I.P> uses an unsupported protocol.
ERR_SSL_VERSION_OR_CIPHER_MISMATCH
I got a Comodo certificate that i requested at my org and got it approved. I have a .key as well which was generated during CSR. I imported the comodo cert into the keystore. Then, I imported both the comodo root cert and .key into truststore. NIFI version is 1.9.2
nifi.properties:
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=mypassword
nifi.security.keyPasswd=
nifi.security.truststore=./conf/truststore.jks
nifi.security.truststoreType=JKS
nifi.security.truststorePasswd=mypassword
nifi.security.user.authorizer=managed-authorizer
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
last few lines of the logs:
2019-07-12 02:29:55,877 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector#45e97963{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
2019-07-12 02:29:55,877 INFO [main] org.eclipse.jetty.server.Server Started #28943ms
2019-07-12 02:29:55,906 INFO [main] org.apache.nifi.nar.NarAutoLoader Starting NAR Auto-Loader for directory ./extensions ...
2019-07-12 02:29:55,907 INFO [main] org.apache.nifi.nar.NarAutoLoader NAR Auto-Loader started
2019-07-12 02:29:55,907 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2019-07-12 02:29:55,907 INFO [main] org.apache.nifi.web.server.JettyServer https://<I.P>:8443/nifi
2019-07-12 02:29:55,907 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:8443/nifi
2019-07-12 02:29:55,909 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2019-07-12 02:29:55,909 INFO [main] org.apache.nifi.NiFi Controller initialization took 19369037824 nanoseconds (19 seconds).
Can you show the output of using the OpenSSL s_client tool to connect to the host? I'm assuming <I.P> is a manual substitution for the actual host IP? Using this version of NiFi, the certificate must have valid SubjectAlternativeName entries for the hostname(s) and IP address(es) you wish to access the service using.
You also want to ensure that the keystore contains the public certificate and private key. The truststore should contain the public certificate and any CA certificates used to sign it (depending on your threshold for desired specificity on accepting incoming certificates for client certificate authentication).

Kafka SSL Authentication Issues for inter-broker communication

I'm currently configuring Apache Kafka with SSL authentication and am coming across an error when starting the service. It appears that the broker starts up correctly(leader election etc seems to occur), but as soon as any cluster operations begin to take place, I get the error below continually in the logs.
[2019-05-16 11:04:00,351] INFO [Controller id=1, targetBrokerId=1] Failed authentication with XXXX/YYYY (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-16 11:04:00,351] DEBUG [Controller id=1, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2019-05-16 11:04:00,351] DEBUG An authentication error occurred in broker-to-broker communication. (org.apache.kafka.clients.ManualMetadataUpdater)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
Tried recreating the key and trust stores, tried dropping SSL from the inter broker listener(this results in an ANONYMOUS principal that I don't want to grant access to any resource).
To explain my configuration:
Running Kafka 2.2 using the SSL principal builder
I have 3 listeners setup - one on a public interface, and two on private interfaces(one for inter-broker comms and one for internal consumers)
SSL is enabled on all 3 listeners
Each listener is tied to it's own key and trust stores(as I need to be able to present different certificates for the internal addresses, as well as being able to trust different signing CA's), and SSL key password is provided for each key/keystore.
Certificates were created using a locally generated key, local CSR generated then signed by a CA running on CFSSL multiroot.
Keystores were then created using the key(same password), signed certificate and CA certificate imported.
Truststore was created and certificate issuing CA(s) added here.
#Kafka Server Properties Configuration
#Broker and listener configuration
broker.id=1
listeners=egress://address1:9093,inter://address1:9094,ingest://address2:9092
advertised.listeners=egress://address1:9093,inter://address1:9094,ingest://address2:9092
listener.security.protocol.map=egress:SSL,inter:SSL,ingest:SSL
inter.broker.listener.name=inter
##
#Listener Trust and Keystore Configurations
#egress configuration
listener.name.egress.ssl.keystore.type=JKS
listener.name.egress.ssl.keystore.location=/data/kafka/pki/egress-keystore.jks
listener.name.egress.ssl.keystore.password=<redacted>
listener.name.egress.ssl.truststore.type=JKS
listener.name.egress.ssl.truststore.location=/data/kafka/pki/egress-truststore.jks
listener.name.egress.ssl.truststore.password=<redacted>
listener.name.egress.ssl.key.password=<redacted>
listener.name.egress.ssl.client.auth=required
listener.name.egress.ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
##
#inter configuration
listener.name.inter.ssl.keystore.type=JKS
listener.name.inter.ssl.keystore.location=/data/kafka/pki/inter-keystore.jks
listener.name.inter.ssl.keystore.password=<redacted>
listener.name.inter.ssl.truststore.type=JKS
listener.name.inter.ssl.truststore.location=/data/kafka/pki/inter-truststore.jks
listener.name.inter.ssl.truststore.password=<redacted>
listener.name.inter.ssl.key.password=<redacted>
listener.name.inter.ssl.client.auth=requested
listener.name.inter.ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
##
#ingest configuration
listener.name.ingest.ssl.keystore.type=JKS
listener.name.ingest.ssl.keystore.location=/data/kafka/pki/ingest-keystore.jks
listener.name.ingest.ssl.keystore.password=<redacted>
listener.name.ingest.ssl.truststore.type=JKS
listener.name.ingest.ssl.truststore.location=/data/kafka/pki/ingest-truststore.jks
listener.name.ingest.ssl.truststore.password=<redacted>
listener.name.ingest.ssl.key.password=<redacted>
listener.name.ingest.ssl.client.auth=required
listener.name.ingest.ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
##
#Generic SSL Configuration
ssl.keystore.type=JKS
ssl.keystore.location=/data/kafka/pki/inter-keystore.jks
ssl.keystore.password=<redacted>
ssl.truststore.type=JKS
ssl.truststore.location=/data/kafka/pki/inter-truststore.jks
ssl.truststore.password=<redacted>
ssl.key.password=<redacted>
ssl.client.auth=requested
ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
ssl.enabled.protocols=TLSv1.2
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=false
super.users=User:<redacted>
##
#General configuration
auto.create.topics.enable=False
delete.topic.enable=True
log.dir=/var/log/kafka
log.retention.hours=24
log.cleaner.enable=True
log.cleanup.policy=delete
log.retention.check.interval.ms=3600000
min.insync.replicas=2
replication.factor=3
default.replication.factor=3
num.partitions=50
offsets.topic.num.partitions=50
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.num.partitions=50
num.replica.fetchers=4
auto.leader.rebalance.enable=True
leader.imbalance.check.interval.seconds=60
transactional.id.expiration.ms=10000
unclean.leader.election.enable=False
zookeeper.connect=zookeeper:2180
zookeeper.session.timeout.ms=100
controlled.shutdown.enable=True
broker.rack=rack1
Did you inserts the certificates to the keystores in the order you described? It could be important to first set the ca, then the certificate signed by the ca to get the chain of trust correctly.

Kafka inter broker SSL handshake failed

I am trying to setup inter-broker SSL (not client) authentication and keep seeing the following errors:
[2019-05-17 06:33:47,151] INFO [Controller id=1004, targetBrokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] INFO [SocketServer brokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] ERROR [Controller id=1004, targetBrokerId=1004] Connection to node 1004 (/$IP:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
My server.properties is:
listeners=PLAINTEXT://$IP:9092,SSL://$IP:9093
security.inter.broker.protocol=SSL
ssl.truststore.password=$PASS
ssl.keystore.password=$PASS
ssl.key.password=$PASS
ssl.endpoint.identification.algorithm=""
ssl.keystore.location=/etc/kafka/kafka.server.keystore.jks
ssl.truststore.location=/etc/kafka/kafka.server.truststore.jks
``
When I run `openssl s_client -debug -connect $IP:9093 -tls1` I get back a list of certificates and `Secure Renegotiation IS supported`
Despite adding `-Djavax.net.debug=all` there's not anything in the logs which points to the problem.
Kafka version 2.2
Any ideas?
I had incorrectly set the value of ssl.endpoint.identification.algorithm="" instead of ssl.endpoint.identification.algorithm", this fixed it.
This value was changed in 2.2 to default to https so setting it to nothing worked.

RabbitMQ Server TLS, client alert: Fatal - Certificate Unknown when starting service

Have RabbitMQ configured to enable TLS with certificates. Key, Cert, and CA defined in .conf file. Upon service startup, error is thrown. Cannot find the cause for this to be thrown and logging isn't giving any more information at the debug level.
Get a client alert failure and am not certain of cause.
2019-03-22 10:04:18.690 [info] <0.7.0> Server startup complete; 4 plugins started.
* rabbitmq_amqp1_0
* rabbitmq_management
* rabbitmq_management_agent
* rabbitmq_web_dispatch
2019-03-22 10:04:24.831 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.690.0>
2019-03-22 10:04:24.831 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.690.0>, {acceptor,{0,0,0,0},5671}) at pid <0.691.0>
2019-03-22 10:04:24.909 [info] <0.688.0> TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown
Our certs didn't have the correct type of X509v3 Extended Key Usage on the cert.
For x509 Auth, you'll need to assign client web auth when creating the certificate.
X509v3 Extended Key Usage:
TLS Web Client Authentication
This won't fix the issue if your certificate CA is broken and can't be verified, but for my issue, this was the resolution.