I am trying to deploy app using activemq and camel, when i am strating my tomcat server I am getting all suspicious messages on console, even when there are no messages in the queue on tye broker , the error messages are
09:56:27.769 [Camel (eventProcessorCamelContext) thread #2 - JmsConsumer[events]] DEBUG o.a.activemq.ActiveMQMessageConsumer - remove: ID:LT-HP-2012907-53495-1360230974137-0:7:1:5, lastDeliveredSequenceId:0
09:56:27.769 [Camel (eventProcessorCamelContext) thread #2 - JmsConsumer[events]] DEBUG org.apache.activemq.ActiveMQSession - ID:LT-HP-xxxxxxxxxx-0:7:1 Transaction Commit :null
09:56:27.769 [Camel (eventProcessorCamelContext) thread #2 - JmsConsumer[events]] DEBUG org.apache.activemq.ActiveMQSession - ID:LT-HP-xxxxxxxx:1 Transaction Rollback
09:56:27.770 [Camel (eventProcessorCamelContext) thread #1 - JmsConsumer[events]] DEBUG o.a.activemq.ActiveMQMessageConsumer - remove: ID:LT-HP-2012907-53495-1360230974137-0:6:1:5, lastDeliveredSequenceId:0
09:56:27.770 [Camel (eventProcessorCamelContext) thread #1 - JmsConsumer[events]] DEBUG org.apache.activemq.ActiveMQSession - ID:LT-HP-xxxxxxxxx:6:1 Transaction Commit :null
09:56:27.770 [Camel (eventProcessorCamelContext) thread #1 - JmsConsumer[events]] DEBUG org.apache.activemq.ActiveMQSession - ID:LT-HP-xxxxxxxxx:6:1 Transaction Rollback
Transaction Commit :null Transaction Rollback ***As you can see in the debug message, there are no messages on the queue, still I am getting debug message as Transaction commit:null and Transaction Rollback. Is it expected? Why ActiveMQ is trying to commit null when there are no messages
Is this expecetd or am i doing anything wrong
Thats verbose logging.
Camel is using Spring DMLC when reading messages transactional from ActiveMQ.
What's essentially is done is something like this (psuedo code, over simplified):
begin transaction()
receive()
.. handle message
commit()
Turn off debug logging if this bothers you.
Related
We have a spring-webflux application running on spring-boot-starter-webflux:jar:2.1.5.RELEASE and reactor-netty:jar:0.8.8.RELEASE. When a reactive client goes away (k8s pod killed or the client subscription is disposed off) before the request is completed, we see the server simply stops processing the request and no more application logs related to that request are seen. However, reactive-netty prints a trace log indicating that the channel is inactive and will be terminated.
Is there anything we can do to handle this termination gracefully? We would ideally like to respond to a cancellation signal from reactor.
2020-03-18T16:46:40.372Z TRACE --- [reactor-http-epoll-2] r.n.c.ChannelOperations : [id: 0x4cad974b, L:/<SERVER_IP_ADDRESS>:8080 ! R:/<SOME_OTHER_IP_ADDRESS>:55436] Disposing ChannelOperation from a channel
java.lang.Exception: ChannelOperation terminal stack
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:391)
at reactor.netty.channel.ChannelOperations.onInboundClose(ChannelOperations.java:360)
at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:72)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420)
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:393)
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:358)
at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1416)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:816)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:834)
The doOnCancel operator allows for a callback to be provided when a subscription is cancelled. This is different from the doOnEach operator which is run only when the subscription completes or errors out.
https://projectreactor.io/docs/core/release/api/reactor/core/publisher/Mono.html#doOnCancel-java.lang.Runnable-
there:
I got the error when use ignite cache.
My system select a master node use zookeeper,and has many slave nodes.The master process ignite cache expired values and put in an ignite queue.The slave node provide data into ignite cache use streamer.addData(k,v) and consume the ignite queue.
My code is:
ignite cache and streamer :
// use zookeeper IpFinder
ignite = Ignition.getOrStart(igniteConfiguration);
igniteCache = ignite.getOrCreateCache(cacheConfiguration);
igniteCache.registerCacheEntryListener(new MutableCacheEntryListenerConfiguration<>(
(Factory<CacheEntryListener<K, CountValue>>)() -> (CacheEntryExpiredListener<K, CountValue>)this
::onCacheExpired, null, true, true));
//onCacheExpired master resolve the expired entry and put in igniteQueue
cacheConfiguration.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.ONE_MINUTE));
igniteDataStreamer = ignite.dataStreamer(igniteCache.getName());
igniteDataStreamer.deployClass(BaseIgniteStreamCount.class);
igniteDataStreamer.allowOverwrite(true);
igniteDataStreamer.receiver(StreamTransformer.from((CacheEntryProcessor<K, CountValue, Object>)(e, arg) -> {
// process the value.
return null;
}));
master process the entry expired from the cache,and put in ignite queue:
CollectionConfiguration collectionConfiguration = new CollectionConfiguration().setCollocated(true);
queue = ignite.queue(igniteQueueName, 0, collectionConfiguration);
the slaves consume the queue.
but i got error log below after running hours later:
2017-09-14 17:06:45,256 org.apache.ignite.logger.java.JavaLogger warning
WARNING: >>> Possible starvation in striped pool.
Thread name: sys-stripe-6-#7%ignite%
Queue: []
Deadlock: false
Completed: 77168
Thread [name="sys-stripe-6-#7%ignite%", id=134, state=WAITING, blockCnt=0, waitCnt=68842]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
at o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:935)
at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:850)
at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$700(CacheContinuousQueryHandler.java:82)
at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1.onEntryUpdated(CacheContinuousQueryHandler.java:413)
at o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryExpired(CacheContinuousQueryManager.java:429)
at o.a.i.i.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:3046)
at o.a.i.i.processors.cache.GridCacheMapEntry.onTtlExpired(GridCacheMapEntry.java:2961)
at o.a.i.i.processors.cache.GridCacheTtlManager$1.applyx(GridCacheTtlManager.java:61)
at o.a.i.i.processors.cache.GridCacheTtlManager$1.applyx(GridCacheTtlManager.java:52)
at o.a.i.i.util.lang.IgniteInClosure2X.apply(IgniteInClosure2X.java:38)
at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1007)
at o.a.i.i.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:198)
at o.a.i.i.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:160)
at o.a.i.i.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:854)
at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1073)
at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
at java.lang.Thread.run(Thread.java:745)
Striped pool is responsible for messages processing. This warning tells you that there is no progress happening on some of the stripes. It may happen due to a bad network connection or when you put massive objects to a cache or a queue.
You may find more information about it in these threads:
http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-td14892.html
http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-message-td15993.html
I have an endpoint that works as a distributor with three other worker endpoints.
The handling endpoint of the received message opens a transaction and tries to import some xml data into an sql db. If some exception is thrown during this process, the exception is caught, the transaction is rolled back and the xml data is written to an error folder.
Simplified, it looks like that:
public void Handle(doSomethingCmd message)
{
System.Data.SqlClient.BeginTransaction();
try
{
//... some xml data import
throw new Exception();
//Commit if succeded
}
catch (Exception exception)
{
System.Data.IDbTransaction.Rollback();
//...Write file to error folder
}
}
In the first place, no retry happens after the transaction rollback. But when the message is sent again, all the worker endpoints (only the workers) get an exception (Cannot enlist transaction, failed to send msg to control queue --> see stacktrace below) and nservicebus does retry the message (this leads to the case, that the file appears several times in the error folder)
It looks like distributed transaction is in an invalid state. I could just handing over the exception (re-throw the exception), so nservicebus handles the rollback for me, but in that case the file is written to the error folder several times as well (due to retry mechanism)
Failed raising finished message processing event.|NServiceBus.Unicast.Queuing.FailedToSendMessageException: Failed to send message to address: someEndpoint.distributor.control#SRVPS01 ---> System.Messaging.MessageQueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 49
--- End of inner exception stack trace ---
at NServiceBus.Transports.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 88
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 75
at NServiceBus.Distributor.MSMQ.ReadyMessages.ReadyMessageSender.SendReadyMessage(String sessionId, Int32 capacityAvailable, Boolean isStarting) in c:\BuildAgent\work\c3100604bbd3ca20\src\NServiceBus.Distributor.MSMQ\ReadyMessages\ReadyMessageSender.cs:line 62
at NServiceBus.Distributor.MSMQ.ReadyMessages.ReadyMessageSender.TransportOnFinishedMessageProcessing(Object sender, FinishedMessageProcessingEventArgs e) in c:\BuildAgent\work\c3100604bbd3ca20\src\NServiceBus.Distributor.MSMQ\ReadyMessages\ReadyMessageSender.cs:line 50
at System.EventHandler1.Invoke(Object sender, TEventArgs e)
at NServiceBus.Unicast.Transport.TransportReceiver.OnFinishedMessageProcessing(TransportMessage msg) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 435
NServicebus version: 4.6.0.0
Queueing: MSMQ
The worker ends it unit of work by sending a message back to the distributor. This send will join the existing distributed transaction. The error you get is caused by this new transactional resource trying to join an already failing transaction. Something has marked the distributed transaction as rolling back.
This is normally caused by your code. Either your database operation is failing somehow or you probably have exceeded the transaction timeout limit handling the message. (Default one minute)
Check your logs to see if you are using above the transaction timeout limit to process the message on the worker.
I am using ORMLite to access a h2 database in Java.
To perform transactions i'm using the static method TransactionManager.callInTransaction.
In case of single independent transactions this works fine. However in case the transactions are nested within each other the inner transactions get committed even if the outer transaction fails.
As in this piece of pseudo-code:
OuterDatabaseTransaction
{
Loop
{
InnerDatabaseTransaction
{
//Multiple update or create statements
//One of the InnerDatabaseTransactions throws a random exception
}
//Alternatively the OuterDatabaseTransaction throws a random
//exception but all commited InnerDatabaseTransactions should rollback still
}
}
So what I would expect is that if any of the inner transactions fails the outer transaction fails too. And if the outer transaction fails none of the inner transactions get committed.
At the moment it seems that each of the inner transactions are committed individually and do not for example share the same Savepoint as the outer transaction.
UPDATE (Thanks dened)
Looking at the trace reveals the following
[TRACE] JdbcDatabaseConnection connection is closed returned false
[TRACE] JdbcDatabaseConnection connection autoCommit is true
[TRACE] JdbcDatabaseConnection connection set autoCommit to false
[DEBUG] TransactionManager had to set auto-commit to false
[TRACE] JdbcDatabaseConnection save-point sp14: id=0 name=ORMLITE15 set with name ORMLITE15
[DEBUG] TransactionManager started savePoint transaction ORMLITE15
[TRACE] JdbcDatabaseConnection connection is closed returned false
[TRACE] JdbcDatabaseConnection connection autoCommit is false
[TRACE] JdbcDatabaseConnection save-point sp15: id=0 name=ORMLITE16 set with name ORMLITE16
[DEBUG] TransactionManager started savePoint transaction ORMLITE16
[TRACE] JdbcDatabaseConnection connection is closed returned false
[TRACE] JdbcDatabaseConnection update statement is prepared and executed returning 1: <...>
[DEBUG] BaseMappedStatement update data with statement <...> changed 1 rows
[TRACE] BaseMappedStatement update arguments: <...>
[TRACE] JdbcDatabaseConnection connection is committed for save-point ORMLITE16
[DEBUG] TransactionManager committed savePoint transaction ORMLITE16
-> [ERROR] TransactionManager after commit exception, rolling back to save-point also threw exception
[TRACE] JdbcDatabaseConnection connection set autoCommit to true
[DEBUG] TransactionManager restored auto-commit to true
[TRACE] JdbcDatabaseConnection connection is closed returned false
Stepping into the source reveals that during rollback of OuterDatabaseTransaction the exception is thrown in the h2 source in org.h2.engine.Session.java within the function below. The reason behind that however I don't understand yet.
private void checkCommitRollback() {
if (commitOrRollbackDisabled && locks.size() > 0) {
throw DbException.get(ErrorCode.COMMIT_ROLLBACK_NOT_ALLOWED);
}
}
UPDATE 2
Posted ORMLite Bug Report
To find the cause you may turn trace level logging on. Working nested transactions should produce the log like this:
TRACE: connection supports save points is true
TRACE: save-point <...> set with name ORMLITE1
DEBUG: started savePoint transaction ORMLITE1
...
TRACE: save-point <...> set with name ORMLITE2
DEBUG: started savePoint transaction ORMLITE2
...
TRACE: connection is committed for save-point ORMLITE2
...
TRACE: save-point <...> set with name ORMLITE3
DEBUG: started savePoint transaction ORMLITE3
...
TRACE: save-point ORMLITE3 is rolled back
...
TRACE: save-point ORMLITE1 is rolled back
in this example ORMLITE1 is save point for outer transaction, ORMLITE2 and ORMLITE3 are for inner transactions. The first inner transaction is initially commited, the second transaction is rolled back to ORMLITE3 and caused rollback of the outer transaction to ORMLITE1, it in turn implicitly cancelled the first inner transaction.
But if you see this in the log:
TRACE: connection supports save points is false
then save points are not supported by the JDBC driver and nested transaction wouldn't work. Theoretically this shouldn't happen as H2 states support of save points.
If you see this:
ERROR: after commit exception, rolling back to save-point also threw exception
then rollback to a save point is failed for some reason. Check you use same ConnectionSource for outer and inner transactions.
Or you my find in the log something else that caused the problem. Also it may be helpful to attach the log to your question. And I suggest to replace the pseudo-code with the real Java code.
Update
Here is a official description of the error you got:
COMMIT_ROLLBACK_NOT_ALLOWED = 90058
The error with code 90058 is thrown when trying to call commit or rollback inside a trigger, or when trying to call a method inside a trigger that implicitly commits the current transaction, if an object is locked. This is not because it would release the lock too early.
Link: http://www.h2database.com/javadoc/org/h2/constant/ErrorCode.html#c90058
May be this will help you further to find the cause of the problem. For me, it is hard to say more not seeing your code. Good luck!
You are correct,
TransactionManager commits each transaction at the end (same behaviour for inner and ounter transaction) as Gray wrote in comments.
The problem is in TransactionManager.java:170, which commits transaction even after inner transaction.
But H2, Mysql, Oracle databases do not support sql COMMIT with savepoint parameter.
From Mysql documentation:
"All savepoints of the current transaction are deleted if you execute a COMMIT, or a ROLLBACK that does not name a savepoint."
From Oracle documentation:
"A simple rollback or commit erases all savepoints. When you roll back to a savepoint, any savepoints marked after that savepoint are erased. The savepoint to which you roll back remains."
There is also comment on this in OrmLite code : JdbcDatabaseConnection.java:94
From my understanding NServiceBus executes the Handle method of an IMessageHandler within a transaction, if an exception propagates out of this method, then NServiceBus will ensure the message is put back on the message queue (up X amount of times before error queue) etc.. so we have an atomic operation so to speak.
Now when if I inside my NServiceBus Message Handle method I do something like this
using(var trans = session.BeginTransaction())
{
person.Age = 10;
session.Update<Person>(person);
trans.Commit()
}
using(var trans2 = session.BeginTransaction())
{
person.Age = 20;
session.Update<Person>(person);
// throw new ApplicationException("Oh no");
trans2.Commit()
}
What is the effect of this on the transaction scope?
Is trans1 now counted as a nested transaction in terms of its relationship with the Nservicebus transaction even though we have done nothing to marry them up? (if not how would one link onto the transaction of NServiceBus?
Looking at the second block (trans2), if I uncomment the throw statement, will the NServiceBus transaction then rollback trans1 as well? In basic scenarios, say I dump the above into a console app, then trans1 is independent, commit, flushed and won't rollback. I'm trying to clarify what happens now we sit in someone else's transaction like NServiceBus?
The above is just example code, im wouldnt be working directly with session, more like through a uow pattern.
If you mark your endpoint as transaction (.MsmqTransport().IsTransactional(true) or just AsA_Server) then the transactions will enlist into the one NServiceBus opened. What this means is that the commits you have inside your handler won't actually happen and the whole thing will either commit or rollback together - unless you specifically tell your transactions not to enlist in the ambient transaction.
Whether or not you work directly with the session or through a UoW, it looks like you want to do more than one for a given message - why? The message is already the natural UoW.