can anyone clarify what the MQ "Prot Algorithm" bit of the ECapFlags3 struct is for and/or does, and why and when it gets set by IBM MQ during a connection setup?
At a high level, we're implementing an SSL offload configuration for a client (Mulesoft) to connect to an IBM MQ queue via an F5. Whenever we enable SSL, the connection fails. With TLS off, the data flow works fine. (it's not something basic, like not sharing an acceptable cipher, etc. - the TLS handshake completes between the Mulesoft client and the F5 fine, and data flows - but the client appears to stop the connection). We don't see any difference in what the F5 sends over to IBM MQ in either case; the only difference we can find is that the IBM MQ replies all have the ECapFlags3 "Prot Algorithm" bit set, when TLS is enabled between the client and the F5.
In performing captures, we note the following:
With TLS on, the TLS handshake completes apparently without issue, and MQ packets do go over to the IBM MQ
With TLS on, we see two "INITIAL_DATA" MQ requests go over, and receive responses back; immediately after that, the client sends a TLS alert
With TLS off, we see the same two INITIAL_DATA messages and responses, then the client sends credentials (i.e., sends a "USERID_DATA" message)
We see no unexpected differences in the first 2 exchanges (the INITIAL_DATA calls) with or without TLS, except that the MQ reply, to both INITIAL_DATA calls, has the msb of the ECapFlags3 set; this bit is labeled as "Prot Algorithm" by Wireshark
The whole exchange happens within 300ms when successful (no TLS), and within less when TLS is enabled, so no timeout should be occurring.
Our working hypothesis is that the "Prot Algorithm" bit being set is having some effect on the client; correction would involve either preventing it from being set, or adjusting the client configuration to handle it cleanly.
Any information on that flag bit - or any other thoughts! - welcome. Thank you!
Related
I have a chat implementation working with CometD.
On front end I have a Client that has a clientId=123 and is talking to VirtualMachine-1
The longpolling connection between the VirtualMachine-1 and the Client is done through the clientId. When the connection is established during the handshake, VirtualMachine-1 registers the 123 clientId as it's own and accepts its data.
For some reason, if VM-1 is restarted or FAILS. The longpolling connection between Client and VM-1 is disconnected (since the VirtualMachine-1 is dead, the heartbeats would fail, thus it would become disconnected).
In which case, CometD loadBalancer will re-route the Client communication to a new VirtualMachine-2. However, since VirtualMachine-2 has different clientId it is not able to understand the "123" coming from the Client.
My question is - what is the cometD behavior in this case? How does it re-route the traffic from VM-1 to a new VM-2 to successfully go through handshaking process?
When a CometD client is redirected to the second server by the load balancer, the second server does not know about this client.
The client will send a /meta/connect message with clientId=123, and the second server will reply with a 402::unknown_session and advice: {reconnect: "handshake"}.
When receiving the advice to re-handshake, the client will send a /meta/handshake message and will get a new clientId=456 from the second server.
Upon handshake, a well written CometD application will subscribe (even for dynamic subscriptions) to all needed channels, and eventually be restored to function as before, almost transparently.
Messages published to the client during the switch from one server to the other are completely lost: CometD does not implement any persistent feature.
However, persisting messages until the client acknowledged them is possible: CometD offers a number of listeners that are invoked by the CometD implementation, and through these listeners an application can persist messages (or other information) into their own choice of persistent (and possibly distributed) store: Redis, RDBMS, etc.
CometD handles reconnection transparently for you - it just takes a few messages between client and the new server.
You also want to read about CometD's in-memory clustering features.
I am using mosquitto_pub to publish the data with TLS using a topic.
I am using mosquitto_sub to subscribe to the topic from mosquitto_pub..
Whenever I fire a mosquitto_pub , I noticed that the wireshark can detect a 3 way handshake each time.
My Question now is, is it possible for mosquitto to do only one time of a 3 way handshake? just to minimize the time of sending the data and receiving it to the other end.
I mean like keep the handshake alive on the first firing of mosquitto_pub, then on the succeeding publishing of message, it will send only the TLS data and not do a 3 way handshake over and over again each time.
What you are describing is SSL/TLS session resumption.
There is support in the mosquitto broker for session resumption, but not in the command line tools.
This is because they would need to store the session id key between each execution. This looks to have been discussed in this mosquitto-dev mailing list thread, but not implemented as there was no demand for it.
You can use TLS session resumption with the Paho C library by settings the Clean Session flag to false (I still think the concept of MQTT session and TLS session should have been kept separate) if the broker supports it.
I loaded a client-side .svclog file inside Microsoft Service Trace Viewer and there are a lot of entries in the log saying setting up secure session and close secure session. On the server side, I can see many instances of trust/RST/SCT/Cancel, indicating that the connections are being closed on the server side, but only after giving a response to a SOAP message. It seems like every web service call involves setting up a TLS session for SOAP, and then the connection being closed immediately after sending a response, requiring that TLS be set up again for the very next call.
I read this article: https://blogs.technet.microsoft.com/tspring/2015/02/23/poor-mans-guide-to-troubleshooting-tls-failures/
It said:
Keep in mind that TCP resets should always be expected at some point as the client closes out the session to the server. However, if there are a high volume of TCP resets with little or no “Application Data” (traffic which contains the encapsulated encrypted data between client and server) then you likely have a problem. Particularly if the server side is resetting the connection as opposed to the client.
Unfortunately, the article doesn't expand on this, because it is exactly what I am seeing!
This is a net.tcp web service installed in some customer environment, set up to use Windows authentication.
What's the next step in my diagnosis?
Most likely the behavior you are seeing is normal, and unless you are experiencing some problems I would not be concerned. The MSFT document you quote is referring to TCP resets, but you said your logs show trust/RST/SCT/Cancel entries, and in that context RST means RequestSecurityToken. In other words, your log messages don't in any way imply that there are TCP reset (RST) frames occurring.
The Web Services Secure Conversation Language (WS-SecureConversation) spec (here) says:
It is not uncommon for a requestor to be done with a security context
token before it expires. In such cases the requestor can explicitly
cancel the security context using this specialized binding based on
the WS-Trust Cancel binding. The following Action URIs are used with
this binding:
http://schemas.xmlsoap.org/ws/2005/02/trust/RST/SCT/Cancel
http://schemas.xmlsoap.org/ws/2005/02/trust/RSTR/SCT/Cancel
Once a
security context has been cancelled it MUST NOT be allowed for
authentication or authorization or allow renewal. Proof of possession
of the key associated with the security context MUST be proven in
order for the context to be cancelled.
If you actually are experiencing transport problems due to unexpected TCP RST frames, or if you are seeing them and are curious to understand their underlying cause, then you'll need to capture network traffic to see how and why TCP resets are occurring, and whether they are normal or abnormal.
I'd do that by firing up WireShark and looking at the frames. If you see FIN, ACK messages from each side then you expect the connection to be closed gracefully after a waiting period. Otherwise you'll see RST frames for a variety of reasons: application resets (performed to avoid tying up a lot of ports in Wait states), bad sequence number when re-accessing a port that's in a Wait state, router or firewall RST messages (typically sent both directions), retransmit timeouts, port choice RST messages, and others.
There are lots of resources to help with TCP traffic analysis. You might find it helpful to take a look at https://blogs.technet.microsoft.com/networking/2009/08/12/where-do-resets-come-from-no-the-stork-does-not-bring-them/ for a quick overview.
If you're not familiar with WireShark it can seem a little complicated, but the thing you want to do here is very simple and you can get your answer very quickly even with no prior experience. Just search for wireshark tutorials and you'll find one that fits your cognitive style.
You can also use WireShark to troubleshoot higher level protocols, including TLS. You can find information about that in many places. I'll just list a few to get you started:
WireShark documentation on SSL is here.
Wikiversity section on HTTPS is here.
A 5-minute youtube tutorial for looking at SSL traffic is here.
I believe this covers your next diagnostic step reasonably well, but if not, feel free to post more information and I can try to provide a better answer.
I'm trying to determine how the heartbeat is negotiated, if at all. It is, after all, called the "requested" heartbeat. For example, if the server heartbeat is set lower than the client heartbeat, the client library should use the minimum, right? Otherwise, all connections will be closed by the server.
The connection documentation doesn't really make it clear and I'm not finding much by searching the code (no conclusive usages of ConnectionConfiguration.RequestedHeartbeat).
The server documentation says
This value is negotiated between the client and RabbitMQ server at the time of connection. The client must be configured to request heartbeats... (although the client can still veto them).
The "official" .NET client library uses the minimum of the client or server heartbeat.
Math.Min(clientValue, serverValue);
I am trying to demonstrate to others that my queue is using SSL, however from the RabbitMQ web management tools there seems to be no distinction over which queues are using SSL and which are not.
Using RabbitMQ management on localhost, I am able to see all my queues. I have set up SSL on port 5671 successfully using the troubleshooting from RabbitMQ website.
Using MassTransit I have configured my incoming bus to use localhost:5671/my_queue_name with a client certificate and all is working successfully - I just can't confirm to others that the queue is secure. If I get a message from the queue using the web management tools, I can read the (JSON) message in plain text. Any ideas how I can prove my messages are secure?
I've attempted using BusDriver to peek the queues but get nothing back (independent of whether is SSL or not).
SSL is used to secure connections, not to encrypt queue contents.
What SSL gives you is that communication from clients to RabbitMQ will be encrypted, so you could theoretically be sure that nobody tampered with your messages.
Also if you need to validate that the sender of the message is a particular user, you could use this RabbitMQ extension: http://www.rabbitmq.com/validated-user-id.html