I'm trying to set up JMS client to use XA transactions with variable timeouts, talking to Active MQ classic (Amazon MQ with ActiveMQ broker 5.15.4)
Calling jmsSession.getXAResource().setTransactionTimeout(timeOut) returns false, indicating that that didn't succeed. This means that sending SESS_XA_SET_TIMEOUT_RESP to the broker resulted in !OK, but I don't see any logs that would explain why (for that matter, so far I can't see any logs from the broker). The same XA logic code works just fine with Artemis.
I found Resource Manager Configuration that says that [default] transaction timeout can be set in the broker configuration file, however it's unclear how exactly this is to be specified, the configuration file schema has no mention of transaction-timeout property, nor can I see any "generic" property list where can it be specified.
So two-part question, really:
How do I configure the "default" transaction timeout mentioned in Resource Manager Configuration
Is there a particular reason I can't change the transaction timeout programmatically.
OK, the short answer - ActiveMQ Classic (5.x) doesn't support distributed transaction timeouts.
I apparently was looking at Artemis's client code, instead of the Classic's. In the Classic client, TransactionContext.setTransactionTimeout() simply returns false:
#Override
public boolean setTransactionTimeout(int seconds) throws XAException {
return false;
}
As far as I can understand, this means that XA transactions never timeout, but I can't find any documentation that talks about any of it.
Related
In ActiveMQ in Action the author states as one of the performance tuning possibilities for ActiveMQ is to set the AsyncSend flag for ActiveMQConnectionFactory. In my case I want to use the JmsPoolConnectionFactory to create a pool of connection and in the same time set the AsyncSend flag. However, I cannot seem to find a way to to that.
You can set jms.useAsyncSend=true on the connection's URL as described in the documentation, e.g.:
tcp://locahost:61616?jms.useAsyncSend=true
Depending on which pooling implementation you are using the way it is created might differ a bit but in general they all just accept an external JMS ConnectionFactory instance and then create JMS resources from there. So this implies that if you want to set configuration that is specific to your own JMS provider then you should be doing that through the ConnectionFactory that you pass to the pool's ConnectionFactory proxy.
I have a spring application which runs in multiple instances on cloudfoundry.
These instances share a database. They have a RabbitListener configured like so:
#RabbitListener(queues = "${items.updated.queue}", exclusive = true)
The queue gets a message if a reimport of items from a certain source is required.
I only want one instance to perform the import. To my understanding this can be accomplished by the exclusive flag.
Now, what would happen if the current exclusive consumer crashes?
Would another currently running instance register itself as the new exclusive consumer? Or does the registration only take place when the application starts up?
Yes, another consumer will be granted access.
Consumers will re-attempt to consume every recoveryInterval milliseconds (default 5000 - 5 seconds).
You can change this by setting the interval or a recoveryBackoff in the listener container.
Note that you will get a WARN log from the container about the failure and an INFO log from the connection factory that the channel was closed due to a failure.
You can either adjust the log levels to reduce these logs, or you can inject custom ConditionalExceptionLogger s into both the container and factory.
See the documentation.
If a consumer fails because one if its queues is being used exclusively, by default, as well as publishing the event, a WARN log is issued. To change this logging behavior, provide a custom ConditionalExceptionLogger in the SimpleMessageListenerContainer's exclusiveConsumerExceptionLogger property. See also the section called “Logging Channel Close Events”.
Whenever I am trying to deploy my application I keep getting this Exception in the logs:
MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis
I couldn't understand why this was happening so I checked domains/domain1/imq/logs/log.txt and this is what I found:
No threads are available to process a new connection on service admin. 10 threads out of a maximum of 10 threads are already in use by other connections. A minimum of 2 threads must be available to process the connection. Please either limit the # of connections or increase the imq.<service>.max_threads property. Closing the new connection. ". Count: service=5 broker=5
Can someone help me with understanding how to increase this count..
I would really appreciate your help on this.
You should change the connection properties (max_threads) of the broker as the error message suggests. The broker configuration file is \domains\\imq\instances\imqbroker\props\config.properties.
This depends on whether you are using OpenMQ in embedded mode or not. If you are using embedded MQ, look for the Thread Pools section of your config in the admin console. One of them will have a max threads set to 10, that will be the one to increase.
It's hard to be sure since you haven't given any other detail from the logs, but that is very likely what you need to change.
Please excuse the Obvious Self-Q/A, but this information is widely misunderstood, and almost always incorrectly answered. So I Wanted to place this information here for people searching for a definitive answer to this problem.
Even so, there's still some information I haven't been able to nail down. I will put this towards the end of the question (skip to that if you are not interested in the preamble).
How do I correctly configure a WCF NetTcp Duplex Reliable Session?
There are many questions and answers regarding this topic, and nearly all of them suggest setting inactivityTimeout="Infinite" in your configuration. This doesn't really seem to work correctly, particularly for the case of NetTcp (It may work correctly for WSDualHttp Bindings, but I have never used those).
There are a number of other issues that are often associated with this: Including, Channel not faulting after client or server unexpectedly disconnected, Channel disconnecting after 10 minutes, Channel randomly disconnecting... Channel throwing exception when trying to open... Unable to configure Metadata on same endpoint...
Please note: There are two concepts that are important below. Infrastructure messages are internal to the way WCF communicates, and are used by the framework to keep things running smoothly. Operation messages are messages that occur because your app has done something, like send a message across the wire. Infrastructure messages are largely invisible to your app (but they still occur in the background) while operation messages are the result of an action your app has taken.
Information I have figured out, through hard won trial and error.
Infinite does not appear to be a valid configuration setting in all situations (and certainly, the visual studio validation schema doesn't know about it).
There are two special configuration converters, called InfiniteIntConverter and InfiniteTimeSpanConverter which will sometimes work to convert the value Infinite to either Int.MaxValue or TimeSpan.MaxValue, but I haven't yet figured out the situations in which this appears to be valid as sometimes it works, and sometimes it doesn't. What's more, it appears that some libraries will allow Infinite in the config, while others will not, so you can succeed in one part of a configuration, but fail in another.
You must configure BOTH inactivityTimeout and receiveTimeout, on both the client and the server. While these values do not HAVE to be the same, they probably should be as they will probably cause confusion if they are not. (technically, you can leave inactivityTimeout to its default value if you want, but you should be aware of its value, and what it does)
inactivityTimeout should NEVER be set to a large value, much less Infinite or TimeSpan.MaxValue.
inactivityTimeout has two functions (and this is not widely understood). The first function defines the maximum amount of time that can elapse on a channel without receiving any "infrastructure" or "operation" messages. The second function defines the time period in which infrastructure messages are sent (half the time specified). If no infrastructure or operation messages have been received during the timeout period, the connection is aborted.
receiveTimeout specifies the maximum amount of time that can elapse between operation messages only. This value can be set to a large value, such as TimeSpan.MaxValue (particularly if your channel runs internally over a trusted network or over a vpn). This value is what defines how long the reliable session will "stay alive" if there is no activity between client and server (other than infrastructure messages). ie, your client does not call any methods of the interface, and your server does not call back into the client.
setting a short inactivityTimeout and a large receiveTimeout keeps your reliable session "tacked up" even when there is no operational activity between your client and server. The short inactivity timeout (i like to keep the default 10 minutes or less) sends infrastructure "ping" messages to keep the TCP connection alive while the long receive timeout keeps the reliable session active. while at the same time providing a reasonable timeout in case of disconnection.
If you set inactivityTimeout to a large value, then the reliable session will not be reliable as it has no way to keep the Tcp connection alive, nor does it have any way to verify the integrity of the connection. It won't know if a user has disconnected unexpectedly until you try and send a message to that client and find out the connection is no longer there. This is why many people who use Infinite for this setting resort to creating a "Ping" method in their service, which is completely unnecessary if you've configured these settings correctly.
If you set inactivityTimeout to a value larger than receiveTimeout then it will likewise also be unreliable, as you will still be governed by the receiveTimeout for operation messages. ie. if you forget to set receiveTimeout and leave it at the default 10 minutes, then if the user is idle for 10 minutes, the connection will be aborted.
When the client or server unexpectedly disconnects (app crashes, network failure, someone trips over the power cord, etc..), the other side may not notice right away. I have attached various ChannelFaulted event handlers in various test situations, and sometimes the connection is faulted right away... other times it doesn't seem to fault at all. What i have discovered through trial and error is that the when it doesn't seem to fault, it will actually fault after the inactivityTimeout expires on that end. (so if it's set to 10 minutes, then after 10 minutes it will call the ChannelFaulted event).
I have not yet figured out why in some situations it notices the disconnection right away, and others it waits for the timer to expire. In both cases, I notice internal first chance communication exceptions thrown and handled by the framework, and there are calls to Abort the connection... but somehow the call to the event handler gets lost and it must wait for the timeout. My suspicion is this is somehow thread related.
When trying to configure Metadata to work across the NetTcp channel, I have had sporadic results. Sometimes it works, sometimes it doesn't. I've read many reports that Metadata doesn't work over NetTcp and that you have to use an Http channel for the Metadata, but I have in fact had it work on several occasions using the net.tcp:// url to generate the proxy. Then I would change something, recompile and it would no longer work. Changing things back, it wouldn't work again. So it was very confusing what magic incantation was necessary to make Metadata function over net.tcp, shared with the endpoint on the same port (obviously with a different address).
When configuring both a NetTcp and Metatdata endpoint on the same service, and specifying non-default settings for connection parameters like listenBacklog, and maxConnections, you also need to make sure the Metadata endpoint uses the same settings, which typically means you have to define a custom binding, since these settings are not available from the standard tcp mex binding. This includes setting listenBacklog and maxPendingConnections on tcpTransport, and groupName and maxOutboundConnectionsPerEndpoint on connectionPoolSettings.
The default setting for the Ordered setting of ReliableSession is True. This uses a lot more overhead than turning it off. If you don't need ordered messages, i would suggest turning it off (need to set this on both sides)
-
Configuration I still need to understand:
How do I correctly configure the shared net.tcp Metadata endpoint? (I will add an example when I get a chance) Currently, i'm specifying an http get url to bypass the problem. It's so inconsistent as to why it sometimes works and sometimes does not. I kept getting the error `The URI Prefix is not recognized' when generating the proxy in Visual Studio.
Why does WCF sometimes Fault the channel immediately upon disconnect, and sometimes waits for inactivityTimeout to expire? What controls/causes one vs the other behavior?
I have a producer sending durable messages to a RabbitMQ exchange. If the RabbitMQ memory or disk exceeds the watermark threshold, RabbitMQ will block my producer. The documentation says that it stops reading from the socket, and also pauses heartbeats.
What I would like is a way to know in my producer code that I have been blocked. Currently, even with a heartbeat enabled, everything just pauses forever. I'd like to receive some sort of exception so that I know I've been blocked and I can warn the user and/or take some other action, but I can't find any way to do this. I am using both the Java and C# clients and would need this functionality in both. Any advice? Thanks.
Sorry to tell you but with RabbitMQ (at least with 2.8.6) this isn't possible :-(
had a similar problem, which centred around trying to establish a channel when the connection was blocked. The result was the same as what you're experiencing.
I did some investigation into the actual core of the RabbitMQ C# .Net Library and discovered the root cause of the problem is that it goes into an infinite blocking state.
You can see more details on the RabbitMQ mailing list here:
http://rabbitmq.1065348.n5.nabble.com/Net-Client-locks-trying-to-create-a-channel-on-a-blocked-connection-td21588.html
One suggestion (which we didn't implement) was to do the work inside of a thread and have some other component manage the timeout and kill the thread if it is exceeded. We just accepted the risk :-(
The Rabbitmq uses a blocking rpc call that listens for a reply indefinitely.
If you look the Java client api, what it does is:
AMQChannel.BlockingRpcContinuation k = new AMQChannel.SimpleBlockingRpcContinuation();
k.getReply(-1);
Now -1 passed in the argument blocks until a reply is received.
The good thing is you could pass in your timeout in order to make it return.
The bad thing is you will have to update the client jars.
If you are OK with doing that, you could pass in a timeout wherever a blocking call like above is made.
The code would look something like:
try {
return k.getReply(200);
} catch (TimeoutException e) {
throw new MyCustomRuntimeorTimeoutException("RabbitTimeout ex",e);
}
And in your code you could handle this exception and perform your logic in this event.
Some related classes that might require this fix would be:
com.rabbitmq.client.impl.AMQChannel
com.rabbitmq.client.impl.ChannelN
com.rabbitmq.client.impl.AMQConnection
FYI: I have tried this and it works.