java.net.SocketException: Broken pipe with JMS/JNDI and MQ series - ssl

Question

I configured WebSphere MQ v6.0.1.1 to be accessed by a Java client using JNDI and JMS via SSL. I tried the client and server on different machine, and on the same machine. I didn't get the same exception on the client side but it's related to a connection problem. On the server side I have nothing in the log.
Different machine client side error: Thread pool thread #0, Exception sending alert: java.net.SocketException: Software caused connection abort: socket write error
*** ServerHelloDone
*** Certificate chain
***
*** ClientKeyExchange, RSA PreMasterSecret, TLSv1
Thread pool thread #0, WRITE: TLSv1 Handshake, length = 141
SESSION KEYGEN:
PreMaster Secret:
0000: 03 01 DB 7F 1B 78 46 24 D1 B3 7F 8F E4 2B 2D 35 .....xF$.....+-5
0010: 1B EB FF C9 01 C9 EC 12 07 0F F9 88 A9 12 45 77 ..............Ew
0020: 22 AE 79 17 C2 9D 4C 97 04 3E BA 91 1F 14 68 44 ".y...L..>....hD
CONNECTION KEYGEN:
Client Nonce:
0000: 50 76 7B FB 0D 45 F0 8D EF 54 E0 AB 2C 3A D4 7D Pv...E...T..,:..
0010: 24 52 FB FB 4F F4 1D E4 CC 2C 4E BA 8B CA 3E 54 $R..O....,N...>T
Server Nonce:
0000: 00 00 00 00 8F 53 C4 4D 2F 2F 41 AA EB 0A 80 2D .....S.M//A....-
0010: D0 E4 51 2A CC BC EE 94 92 BD CD E0 9B C9 EB 3D ..Q*...........=
Master Secret:
0000: 9D 93 ED F3 8A 97 39 7F 71 5F 34 52 30 A6 8E 38 ......9.q_4R0..8
0010: BC 17 59 28 78 63 AA 66 63 D0 EE 1C C6 54 CA D1 ..Y(xc.fc....T..
0020: F2 F0 ED 7E D7 81 33 C6 E3 1B 7C 46 C0 FB C8 5C ......3....F...\
Client MAC write Secret:
0000: 57 56 3D 05 B1 27 BE 56 A8 FD 07 64 0A 96 62 F2 WV=..'.V...d..b.
0010: AE AF B5 98 ....
Server MAC write Secret:
0000: F5 C7 B2 D2 79 11 90 6C C8 FD 86 8B E5 AE 59 71 ....y..l......Yq
0010: B2 A7 AB D3 ....
Client write key:
0000: 54 FD FD 8B C2 B4 8B 3F 38 23 25 5A 8A 41 26 9B T......?8#%Z.A&.
Server write key:
0000: 6D 9C C0 97 ED 21 3F 0E 0A FB E2 2B EE C0 5F 01 m....!?....+.._.
... no IV used for this cipher
Thread pool thread #0, WRITE: TLSv1 Change Cipher Spec, length = 1
*** Finished
verify_data: { 182, 85, 56, 238, 250, 233, 155, 119, 224, 254, 23, 196 }
***
Thread pool thread #0, WRITE: TLSv1 Handshake, length = 36
Thread pool thread #0, READ: TLSv1 Change Cipher Spec, length = 1
Thread pool thread #0, READ: TLSv1 Handshake, length = 36
*** Finished
verify_data: { 215, 140, 30, 150, 82, 161, 85, 160, 127, 189, 226, 74 }
***
%% Cached client session: [Session-1, SSL_RSA_WITH_RC4_128_SHA]
Thread pool thread #0, setSoTimeout(120000) called
Thread pool thread #0, WRITE: TLSv1 Application Data, length = 150
Thread pool thread #0, READ: TLSv1 Application Data, length = 56
Thread pool thread #0, WRITE: TLSv1 Application Data, length = 48
Thread pool thread #0, called close()
Thread pool thread #0, called closeInternal(true)
Thread pool thread #0, SEND TLSv1 ALERT: warning, description = close_notify
Thread pool thread #0, WRITE: TLSv1 Alert, length = 22
Thread pool thread #0, Exception sending alert: java.net.SocketException: Software caused connection abort: socket write error
Same machine client side error: Thread pool thread #0, Exception sending alert: java.net.SocketException: Broken pipe
It seems that the client writes whereas the server has already closed the connection.
EDIT:
10/10/12 2:26:23 PM - Process(10995.12) User(mqm) Program(amqrmppa)
AMQ9631: The CipherSpec negotiated during the SSL handshake does not match the
required CipherSpec for channel 'SSL.CHANNEL'.
EXPLANATION:
There is a mismatch between the CipherSpecs on the local and remote ends of
channel 'SSL.CHANNEL'. The channel will not run until this mismatch is
resolved. The CipherSpec required in the local channel definition is
'RC4_SHA_US'. The name of the CipherSpec negotiated during the SSL handshake is
'RC4_SHA_US'. A code is displayed if the name of the negotiated CipherSpec
cannot be determined.
ACTION:
Change the channel definitions for 'SSL.CHANNEL' so the two ends have matching
CipherSpecs and restart the channel. If the certificate in use by one end of
the channel is a Global Server Certificate, then the negotiated CipherSpec may
not match that specified on either end of the channel. This is because the SSL
protocol allows a Global Server Certificate to automatically negotiate a higher
level of encryption. In these cases specify a CipherSpec which meets the
requirements of the Global Server Certificate.
----- amqccisa.c : 851 --------------------------------------------------------
10/10/12 2:26:23 PM - Process(10995.12) User(mqm) Program(amqrmppa)
AMQ9999: Channel program ended abnormally.
EXPLANATION:
Channel program 'SSL.CHANNEL' ended abnormally.
ACTION:
Look at previous error messages for channel program 'SSL.CHANNEL' in the error
files to determine the cause of the failure.
----- amqrmrsa.c : 468 --------------------------------------------------------
Edit 2:
1 : DIS CHANNEL(SSL.CHANNEL) SSLCIPH
AMQ8414: Display Channel details.
CHANNEL(SSL.CHANNEL) CHLTYPE(SVRCONN)
SSLCIPH(RC4_SHA_US)
The Cipher used client side using JMSAdmin:
DEFINE QCF(QCF_NAME) SYNCPOINTALLGETS(YES) HOSTNAME(HOST) PORT(1414) TRANSPORT(client) QMANAGER(MYQMGR) CHANNEL(SSL.CHANNEL) SSLCIPHERSUITE(SSL_RSA_WITH_RC4_128_SHA)
Base on SSL CipherSpecs and CipherSuites RC4_SHA_US seems to match SSL_RSA_WITH_RC4_128_SHA.

Answer 1

There is a possibility when running the client on the same host as the QMgr to connect using bindings mode (shared memory) rather than over the network stack. Since bindings mode connections do not use the network stack, SSL would make no difference.
Assuming that the client is connecting over the network in both cases, there's nothing about the location of the client on one server or another that would influence the SSL connection other than that the client configurations were different from one instance to the other. The client is still going through the network stack and presenting a connection request to the QMgr's listener. The client finds its keystore the same way and all the QMgr sees is a connection request presented to the listener. So if you are getting different results between the two client instances, look for configuration discrepancies.
My method for debugging SSL connections on WMQ is to progress through the following sequence making sure each step works before advancing to the next:
Get the channel running without SSL. This validates that the channel names are spelled correctly, that a network route exists between the endpoints, that the QMgr's listener is running and that the client points to the right port.
Get the channel running with the SVRCONN definition set to SSLCAUTH(OPTIONAL). This performs an anonymous SSL connection similar to what your browser does. The QMgr presents a certificate to the client but does not request one back. This validates that the QMgr can find its certificate and that the client can find its trust store and properly validates the QMgr's cert.
Set the SVRCONN channel to SSLCAUTH(REQUIRED). This now requires the client to find its keystore (in the last step it required only its trust store) and to be able to find its certificate. It also requires the QMgr to be able to validate the client's cert.
The difference between the last two steps helps to isolate the problem by testing the SSL credential exchange in only one direction at a time.
You mention that there's "nothing in the log" when this happens. Which log? There are two sets of error logs. One is the server-global log at {WMQ home}/errors and the other is the QMgr error log at {WMQ home}/QMgrs/{QMgr name}/errors. Error log entries are made to the QMgr's error log when the MQ can identify the QMgr associated with the error. However, when an SSL connection is requested, MQ does not know which QMgr the connection has requested until after the SSL connection has completed. Because of this, many SSL negotiation errors on the server side are reported in the global error log.
I'd suggest running through the steps outlined above to determine which SSL credential exchange is failing and then looking for the error log entries in both the QMgr and global error log files. If this does not resolve the problem, please update your question noting the step in the process that fails and any error log entries identified by which log they were found in.
Also, V6 of MQ went out of service as of last month. Moving to a supported version of WMQ client and server would allow you to open a PMR, would provide much better Java/JMS performance, and will allow you to use more secure ciphers such as SHA-2 hashes and the new elliptic curve crypto supported by GSKit 8. Since WMQ V6 is out of support, at most only one additional Fix Pack will be released, after which security vulnerabilities in that version will not be addressed. If you are using SSL, I assume security is somewhat important and that you would want to use a version that will receive fixes if a new vulnerability is discovered.
UPDATE
Responding to the question update regarding the Global Server Certificate, it is necessary to understand how WMQ implements SSL/TLS. When the connection is made, the TLS negotiation (if you specify an SSL , WMQ performs a TLS session using an SSL cipher) follows the spec and the spec allows for a negotiation of the ciphersuite. When a global server certificate is used, the cert can specify a minimum acceptable cipher strength and this influences the negotiation.
When the TLS session is completed successfully, the connection is then handed to WebSphere MQ. Only then does the QMgr check the channel parameters such as "what QMgr is the connection requested for?" or, more importantly, "does the connection's negotiated cipherspec match the channel definition?" Typically it fails based on a mismatch between client and server. With a global server certificate it can fail because of a mismatch between the negotiated cipherspec versus the minimum acceptable as specified by the certificate.
What the error message is saying is that it is possible for the ciphersuite specified by the client to exactly match the cipherspec specified in the channel and still fail to connect because the global server certificate specifies a minimum cipher strength that is greater than that used by the client and the QMgr. There is more on this at Cipherspec mismatches in the Infocenter but in this case the error message is almost as informative as the Infocenter.