I am stuck at using SSL in IBM Websphere MQ (9.2).
I am building a client library for MQ and to get more familiar with MQ on the server side I have installed IBM MQ Developer edition and ran the supplied scripts to create a 'default' MQ server instance.
Created an client connection for the DEV.APP.SVRCONN server connection
Created a personal certificate by using the IBM Key management tool and named it ibmwebspheremq
Enabled SSL on the Queue Manager (QM1) and labelled it ibmwebspheremq
Updated the SSL configuration for the DEV.APP.SVRCONN channel and set the cipherspec property to TLS 1.2, 256-bit Secure Hash Algorithm, 128-bit AES encryption (TLS_RSA_WITH_AES_128_CBC_SHA256) and made SSL required.
Tested my settings with:
amqssslc -l ibmwebspheremq -k C:\ProgramData\IBM\MQ\qmgrs\QM1\ssl\key -c DEV.APP.SVRCONN -x 127.0.0.1 -s TLS_RSA_WITH_AES_128_CBC_SHA256 -m QM1
And that gave me:
Sample AMQSSSLC start
Connecting to queue manager QM1
Using the server connection channel DEV.APP.SVRCONN
on connection name 127.0.0.1.
Using SSL CipherSpec TLS_RSA_WITH_AES_128_CBC_SHA256
Using SSL key repository stem C:\ProgramData\IBM\MQ\qmgrs\QM1\ssl\key
Certificate Label: ibmwebspheremq
No OCSP configuration specified.
MQCONNX ended with reason code 2035
Error details (from log):
The active values of the channel were 'MCAUSER(app) CLNTUSER(Wilko)
SSLPEER(SERIALNUMBER=61:9B:A4:3E,CN=DESKTOP-ROH98N2,C=NL)
SSLCERTI(CN=DESKTOP-ROH98N2,C=NL) ADDRESS(DESKTOP-ROH98N2)'. The
MATCH(RUNCHECK) mode of the DISPLAY CHLAUTH MQSC command can be used to
identify the relevant CHLAUTH record.
ACTION:
Ensure that the application provides a valid user ID and password, or change
the queue manager connection authority (CONNAUTH) configuration to OPTIONAL to
allow client applications to connect which have not supplied a user ID and
password.
----- cmqxrmsa.c : 2086 -------------------------------------------------------
22/11/2021 15:51:37 - Process(15880.45) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(DESKTOP-ROH98N2) Installation(Installation1)
VRMF(9.2.3.0) QMgr(QM1)
Time(2021-11-22T14:51:37.594Z)
CommentInsert1(DEV.APP.SVRCONN)
CommentInsert2(15880(1112))
CommentInsert3(127.0.0.1)
AMQ9999E: Channel 'DEV.APP.SVRCONN' to host '127.0.0.1' ended abnormally.
EXPLANATION:
The channel program running under process ID 15880(1112) for channel
'DEV.APP.SVRCONN' ended abnormally. The host name is '127.0.0.1'; in some cases
the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 630 --------------------------------------------------------
I am kind of stuck, I also saw in the log that there is PEER related info dumped, but I am not sing the SSLPEER settings (I just want to let everyone connect with the same certificate).
EDIT 2:
Output from RUNMQSC QM1 and command DISPLAY QMGR CONNAUTH:
1 : DISPLAY QMGR CONNAUTH
AMQ8408I: Display Queue Manager details.
QMNAME(QM1) CONNAUTH(DEV.AUTHINFO)
Output from RUNMQSC QM1 and command DISPLAY AUTHINFO(name-from-previous-command):
3 : DISPLAY AUTHINFO(DEV.AUTHINFO)
AMQ8566I: Display authentication information details.
AUTHINFO(DEV.AUTHINFO) AUTHTYPE(IDPWOS)
ADOPTCTX(YES) DESCR( )
CHCKCLNT(REQDADM) CHCKLOCL(OPTIONAL)
FAILDLAY(1) AUTHENMD(OS)
ALTDATE(2021-11-18) ALTTIME(15.09.20)
Output from DISPLAY CHLAUTH(*):
4 : DISPLAY CHLAUTH(*)
AMQ8878I: Display channel authentication record details.
CHLAUTH(DEV.ADMIN.SVRCONN) TYPE(USERMAP)
CLNTUSER(admin) USERSRC(CHANNEL)
AMQ8878I: Display channel authentication record details.
CHLAUTH(DEV.ADMIN.SVRCONN) TYPE(BLOCKUSER)
USERLIST(nobody)
AMQ8878I: Display channel authentication record details.
CHLAUTH(DEV.APP.SVRCONN) TYPE(ADDRESSMAP)
ADDRESS(*) USERSRC(CHANNEL)
CHCKCLNT(REQUIRED)
AMQ8878I: Display channel authentication record details.
CHLAUTH(SYSTEM.ADMIN.SVRCONN) TYPE(ADDRESSMAP)
ADDRESS(*) USERSRC(CHANNEL)
AMQ8878I: Display channel authentication record details.
CHLAUTH(SYSTEM.*) TYPE(ADDRESSMAP)
ADDRESS(*) USERSRC(NOACCESS)
I was expecting not having to provide username and password when using certificates. What am I missing here?
Your queue manager is configured to mandate passwords for any client connections that are trying to run with a resolved MCAUSER that is privileged. That is what CHCKCLNT(REQDADM) on your AUTHINFO(DEV.AUTHINFO) does.
In addition, your CHLAUTH rule for the DEV.APP.SVRCONN channel has upgraded this further to mandate passwords for ALL connections using that channel.
If your intent is to have channels that supply a certificate not be subject to this mandate, then you should add a further, more specific, CHLAUTH rule, something along these lines:-
SET CHLAUTH(DEV.APP.SVRCONN) TYPE(SSLPEERMAP) +
SSLPEER('SERIALNUMBER=61:9B:A4:3E,CN=DESKTOP-ROH98N2,C=NL') +
SSLCERTI('CN=DESKTOP-ROH98N2,C=NL') CHCKCLNT(ASQMGR) USERSRC(CHANNEL)
Bear in mind that if this connection is asserting a privileged user id, it will still be required to supply a password from the system-wide setting of CHCKCLNT(REQDADM).
Remember, if you are ever unsure which CHLAUTH rule you are matching against, all those details you saw in the error message can be used to form a DISPLAY CHLAUTH command to discover exactly which rule you have matched. Read more about that in I’m being blocked by CHLAUTH – how can I work out why?
I'm currently configuring Apache Kafka with SSL authentication and am coming across an error when starting the service. It appears that the broker starts up correctly(leader election etc seems to occur), but as soon as any cluster operations begin to take place, I get the error below continually in the logs.
[2019-05-16 11:04:00,351] INFO [Controller id=1, targetBrokerId=1] Failed authentication with XXXX/YYYY (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-16 11:04:00,351] DEBUG [Controller id=1, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2019-05-16 11:04:00,351] DEBUG An authentication error occurred in broker-to-broker communication. (org.apache.kafka.clients.ManualMetadataUpdater)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
Tried recreating the key and trust stores, tried dropping SSL from the inter broker listener(this results in an ANONYMOUS principal that I don't want to grant access to any resource).
To explain my configuration:
Running Kafka 2.2 using the SSL principal builder
I have 3 listeners setup - one on a public interface, and two on private interfaces(one for inter-broker comms and one for internal consumers)
SSL is enabled on all 3 listeners
Each listener is tied to it's own key and trust stores(as I need to be able to present different certificates for the internal addresses, as well as being able to trust different signing CA's), and SSL key password is provided for each key/keystore.
Certificates were created using a locally generated key, local CSR generated then signed by a CA running on CFSSL multiroot.
Keystores were then created using the key(same password), signed certificate and CA certificate imported.
Truststore was created and certificate issuing CA(s) added here.
#Kafka Server Properties Configuration
#Broker and listener configuration
broker.id=1
listeners=egress://address1:9093,inter://address1:9094,ingest://address2:9092
advertised.listeners=egress://address1:9093,inter://address1:9094,ingest://address2:9092
listener.security.protocol.map=egress:SSL,inter:SSL,ingest:SSL
inter.broker.listener.name=inter
##
#Listener Trust and Keystore Configurations
#egress configuration
listener.name.egress.ssl.keystore.type=JKS
listener.name.egress.ssl.keystore.location=/data/kafka/pki/egress-keystore.jks
listener.name.egress.ssl.keystore.password=<redacted>
listener.name.egress.ssl.truststore.type=JKS
listener.name.egress.ssl.truststore.location=/data/kafka/pki/egress-truststore.jks
listener.name.egress.ssl.truststore.password=<redacted>
listener.name.egress.ssl.key.password=<redacted>
listener.name.egress.ssl.client.auth=required
listener.name.egress.ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
##
#inter configuration
listener.name.inter.ssl.keystore.type=JKS
listener.name.inter.ssl.keystore.location=/data/kafka/pki/inter-keystore.jks
listener.name.inter.ssl.keystore.password=<redacted>
listener.name.inter.ssl.truststore.type=JKS
listener.name.inter.ssl.truststore.location=/data/kafka/pki/inter-truststore.jks
listener.name.inter.ssl.truststore.password=<redacted>
listener.name.inter.ssl.key.password=<redacted>
listener.name.inter.ssl.client.auth=requested
listener.name.inter.ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
##
#ingest configuration
listener.name.ingest.ssl.keystore.type=JKS
listener.name.ingest.ssl.keystore.location=/data/kafka/pki/ingest-keystore.jks
listener.name.ingest.ssl.keystore.password=<redacted>
listener.name.ingest.ssl.truststore.type=JKS
listener.name.ingest.ssl.truststore.location=/data/kafka/pki/ingest-truststore.jks
listener.name.ingest.ssl.truststore.password=<redacted>
listener.name.ingest.ssl.key.password=<redacted>
listener.name.ingest.ssl.client.auth=required
listener.name.ingest.ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
##
#Generic SSL Configuration
ssl.keystore.type=JKS
ssl.keystore.location=/data/kafka/pki/inter-keystore.jks
ssl.keystore.password=<redacted>
ssl.truststore.type=JKS
ssl.truststore.location=/data/kafka/pki/inter-truststore.jks
ssl.truststore.password=<redacted>
ssl.key.password=<redacted>
ssl.client.auth=requested
ssl.principal.mapping.rules=RULE:^.*[Oo][Uu]=([a-zA-Z0-9.-]*).*$/$1/L,DEFAULT
ssl.enabled.protocols=TLSv1.2
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=false
super.users=User:<redacted>
##
#General configuration
auto.create.topics.enable=False
delete.topic.enable=True
log.dir=/var/log/kafka
log.retention.hours=24
log.cleaner.enable=True
log.cleanup.policy=delete
log.retention.check.interval.ms=3600000
min.insync.replicas=2
replication.factor=3
default.replication.factor=3
num.partitions=50
offsets.topic.num.partitions=50
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.num.partitions=50
num.replica.fetchers=4
auto.leader.rebalance.enable=True
leader.imbalance.check.interval.seconds=60
transactional.id.expiration.ms=10000
unclean.leader.election.enable=False
zookeeper.connect=zookeeper:2180
zookeeper.session.timeout.ms=100
controlled.shutdown.enable=True
broker.rack=rack1
Did you inserts the certificates to the keystores in the order you described? It could be important to first set the ca, then the certificate signed by the ca to get the chain of trust correctly.
Have RabbitMQ configured to enable TLS with certificates. Key, Cert, and CA defined in .conf file. Upon service startup, error is thrown. Cannot find the cause for this to be thrown and logging isn't giving any more information at the debug level.
Get a client alert failure and am not certain of cause.
2019-03-22 10:04:18.690 [info] <0.7.0> Server startup complete; 4 plugins started.
* rabbitmq_amqp1_0
* rabbitmq_management
* rabbitmq_management_agent
* rabbitmq_web_dispatch
2019-03-22 10:04:24.831 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.690.0>
2019-03-22 10:04:24.831 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.690.0>, {acceptor,{0,0,0,0},5671}) at pid <0.691.0>
2019-03-22 10:04:24.909 [info] <0.688.0> TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown
Our certs didn't have the correct type of X509v3 Extended Key Usage on the cert.
For x509 Auth, you'll need to assign client web auth when creating the certificate.
X509v3 Extended Key Usage:
TLS Web Client Authentication
This won't fix the issue if your certificate CA is broken and can't be verified, but for my issue, this was the resolution.
I am trying to establish secure connection between Openfire and smack client and get the error
org.jivesoftware.smack.SmackException$SecurityRequiredByClientException:
SSL/TLS required by client but not supported by server
I am using Openfire 4.0.1 and have enabled 5223. Using Smack on client side and connection configuration is as follows:
configBuilder.setSecurityMode(ConnectionConfiguration.SecurityMode.required);
configBuilder.setServiceName(service_name);
configBuilder.setHost(server);
configBuilder.setPort(5223);
setHttpsClient(); // Sets a SSL context
configBuilder.setCustomSSLContext(mSSLContext);
configBuilder.setSocketFactory(mSSLContext.getSocketFactory());
connection = new XMPPTCPConnection( configBuilder.build( ) );
connection.connect();
After initial exchange server sends
<stream:features><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
<mechanism>PLAIN</mechanism><mechanism>ANONYMOUS</mechanism>
<mechanism>SCRAM-SHA-1</mechanism><mechanism>CRAM-MD5</mechanism>
<mechanism>DIGEST-MD5</mechanism></mechanisms><compression
xmlns="http://jabber.org/features/compress"><method>zlib</method>
</compression><auth xmlns="http://jabber.org/features/iq-auth"/><register
xmlns="http://jabber.org/features/iq-register"/></stream:features>
which does not contain starttls.
I had set up a 6-node Cassandra cluster spanning two AWS regions / datacenters (3 in each) and everything was working fine. After getting that much working I attempted to enable internode encryption which I cannot get to work properly, despite reading innumerable documents on the subject and fiddling endlessly.
I don't see any errors or anything out of the ordinary in the logs. I do see the following line in the logs which indicates it has started the encrypted messaging service, as expected:
MessagingService.java:482 - Starting Encrypted Messaging Service on SSL port 7001
I have enabled verbose logging for SSL in cassandra-env.sh, however this does not produce any errors or additional information about SSL internode connections that I can see (update below):
JVM_OPTS="$JVM_OPTS -Djavax.net.debug=ssl"
I can connect to from one node to all the others on the encrypted messaging port 7001 using nc, so there's no firewall issue.
ubuntu#ip-5-6-7-8:~$ nc -v 1.2.3.4 7001
Connection to 1.2.3.4 7001 port [tcp/afs3-callback] succeeded!
I can connect to each node locally using cqlsh (I haven't enabled client-server encryption) and can query the system keyspace, etc.
However, if I run nodetool status I see that the nodes cannot see each other. Only the node that I'm querying the cluster on is present in the list. This was not the case before internode encryption was enabled, they could all see each other just fine then.
ubuntu#ip-5-6-7-8:~$ nodetool status
Datacenter: us-east_A
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 1.2.3.4 144.75 KB 256 ? 992ae1bc-77e4-4ab1-a18f-4db62bb0ce6f 1b
My process was this:
Created a certificate authority for my cluster
Created a keystore and truststore for each node and added my CA certificate chain to both
Generated a key pair and CSR for each node, signed it with my CA, and added the resulting certificate to each node's keystore
Updated each node's configuration as reads below
Restarted all nodes
The server encryption configuration I'm using is this, with the appropriate values in the $variables.
server_encryption_options:
internode_encryption: all
keystore: $keystore_path
keystore_password: $keystore_passwd
truststore: $truststore_path
truststore_password: $truststore_passwd
require_client_auth: true
protocol: TLS
algorithm: SunX509
store_type: JKS
cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
If anybody could offer some insight or a direction to look in it would be greatly appreciated.
Update: Cipher Suite Agreement
Apparently SSL debug logging prints to stdout, which is not logged to Cassandra's logfiles, so I didn't see that output before. Running Cassandra in the foreground I can see a ton of SSL errors tracing out, all of which complain of handshake failure, because:
javax.net.ssl.SSLHandshakeException: no cipher suites in common
In an attempt to solve this problem I have switched to the Oracle JRE (I was being lazy and using OpenJDK before) and installed the JCE unlimited strength cryptography policy files to ensure all possible ciphers would be supported.
It didn't fix anything.
This is especially confusing given that all these nodes are exactly identical: hardware, OS vendor and version, Java vendor and version, Cassandra version, and configuration file. I cannot imagine why they cannot agree on a cipher suite under these circumstances.
The following is the full error that is traced:
*** ClientHello, TLSv1.2
RandomCookie: GMT: 1449074039 bytes = { 205, 93, 27, 38, 184, 219, 250, 8, 232, 46, 117, 84, 69, 53, 225, 16, 27, 31, 3, 7, 203, 16, 133, 156, 137, 231, 238, 39 }
Session ID: {}
Cipher Suites: [TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 0 }
***
%% Initialized: [Session-3, SSL_NULL_WITH_NULL_NULL]
%% Invalidated: [Session-3, SSL_NULL_WITH_NULL_NULL]
ACCEPT-/1.2.3.4, SEND TLSv1.2 ALERT: fatal, description = handshake_failure
ACCEPT-/1.2.3.4, WRITE: TLSv1.2 Alert, length = 2
ACCEPT-/1.2.3.4, called closeSocket()
ACCEPT-/1.2.3.4, handling exception: javax.net.ssl.SSLHandshakeException: no cipher suites in common
ACCEPT-/1.2.3.4, called close()
ACCEPT-/1.2.3.4, called closeInternal(true)
INFO 16:33:59 Waiting for gossip to settle before accepting client requests...
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
ACCEPT-/1.2.3.4, setSoTimeout(10000) called
ACCEPT-/1.2.3.4, READ: SSL v2, contentType = Handshake, translated length = 57
After a great deal more poking and prodding I've finally managed to get this to work. The problem was related to certificates and the keystore.
As a result of these problems the SSL handshake would fail either due to certificate chain problems or cipher suite agreement problems. Cassandra rather unhelpfully discards errors related to SSL and logs nothing.
In any case, I managed to get things working by doing the following:
Ensure that the CA generates node certificates with both client and server key usage attributes. Failing to include one or the other will prevent nodes from authenticating to each other properly. This presents itself as the cipher suite agreement error. If you're using OpenSSL to manage your CA, I've included the -extensions configuration I used below.
Ensure that both the root and any intermediate CA certificates you are using (if you're using an intermediary CA) are imported into both the keystore and truststore.
Ensure that the node certificate imported into the keystore includes the full trust chain from the primary certificate down to the CA root, including any intermediaries – even though you have already imported these CA certificates separately into the keystore. Failing to do this presents itself as an invalid certificate chain errors.
OpenSSL CA Config
Here's my extensions section for dual-role client/server certificates. You can include this in your OpenSSL config file and reference it when signing by specifying -extensions dual_cert.
[ dual_cert ]
# Extensions for dual-role user/server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "Client/Server Dual-role Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
Creating a PEM containing the full trust chain
To create a single PEM file which contains the full trust chain for your node certificate, simply cat all the certificate files in reverse order from the node certificate down to the CA root.
cat node1.crt ca-intermediate.crt ca-root.crt > node1-full-chain.crt