While developing application in Moqui framework(1.4.1), a frustrating issue regarding the bitronix transaction timeout occurs. I am not able to understand the reason why this happens and the only solution to this is that I have to restart my system.
I would really like to know how can I rectify this problem.
The exception is like this
Setting the transaction timeout is done where the transaction is begun. This is in your code written using the Moqui tools such as a service or screen, or Java/Groovy/etc code that uses the Moqui TransactionFacade or the JTA interfaces directly.
By default Moqui screens are not run in transactions, unless you set the screen.#begin-transaction attribute to do so. Chances are your problem is in a long-running service, and by default Moqui services ARE run in transactions. Set the timeout using the service.#transaction-timeout attribute on the service that beings the transaction. By default services use a transaction already in place if there is one, so this needs to be on the outer most service where the transaction is actual begun.
For more details about services and transaction management see the Making Apps with Moqui book, available for download from moqui.org.
You may have another issue in your code and that is the socket timing out for the browser request (I see that as well in the log in your screenshot). There are some ways around this, but also some things you can't control so easily like when the browser times out. For a good UI it is also best not have your user sit and wait more than the typical 30-60s where such timeouts start hitting. Change your code to run in the background and if needed add something to your screen to monitor status and/or progress of the job.
If you are loading large files with the java -jar load command, you can use the timeout parameter to set the timeout in seconds to lets say 3600, because the default is 600 seconds.
More about the load command parameters at:
java -jar moqui.war help
Related
The application is using Apache Server as a web server and Tomcat as an application server.
Operations/requests can be triggered from the UI, which can take time to return from the server as it does some processing like fetching data from the database and performing calculations on that data. This time depends on the amount of data in the database and the duration of data it is processing. It could be as long as 30min to an hour or 2 min's based on the parameters.
Apart from this, there are some other calls which fetche small amount of data from the database and return immediately.
Now when I have multiple, say 4 or 5 of these long heavy calls to the server, and they are currently running, when I make a call that is supposed to be smaller and return immediately, this call also hangs as it never reaches my controller.
I am unable to find a way to debug this issue or find a resolution. Please let me know if in case you happen to know how to proceed with this issue.
I am using Spring, with c3p0 connection pooling with Hibernate.
So I figured out what was wrong with the application, and thought about sharing it in case someone somewhere faces the same issue. It turns out nothing was wrong with the application server or the web server, when technically speaking it was the browsers fault.
I found out that the browser can only have a limited number of open concurrent calls to a domain. In the case of the latest version of chrome at the time of writing is 6. This is something all the browsers do to prevent DDOS attacks.
As in my application, the HTTP calls take a lot of time to return until the calculations are completed several HTTP calls accumulate concurrently and as a result, the browser stops sending any further calls after the 6th concurrent call and it feels like the application is unresponsive. You can read about the maximum no of concurrent calls by a browser in SO.
A possible solution I have thought is either polling or even better Long Polling. I would have used WebSockets but then we would need to make a lot of changes.
We have a number of .NET applications that make use of log4net for logging to a sql server databsae. For various reasons (unrelated to log4net...I think) on occasion logging stops. The application might continue working but logging will not continue until the IIS application pool is recycled. The obvious solution would be to add reconnectonerror to the log4net appender. However, as I understand it it is always suggested that "connect timeout=1" be added to the appended connection string. Why?
What I mean is...
If log4net logging worked without "connect timeout=1" why would including make adding "connect timeout=1" matter?
According to the documentation, the act of reconnecting may block the calling thread.
More specifically, if a connection is not available, Log4Net will attempt to reconnect once there are enough messages for a batch. If there is a chronic problem with the database, this could result in performance degradation -- especially if you have configured lots of logging or a small batch size.
One of Apache's design goals is to allow log statements to remain in production code without incurring high performance cost. That's where the connect timeout suggestion comes from. If you have to pay the cost of reconnecting, at least make it quick so you don't take too much of a performance hit.
Sources :
http://logging.apache.org/log4net/release/sdk/log4net.Appender.AdoNetAppender.ReconnectOnError.html
http://mail-archives.apache.org/mod_mbox/logging-log4net-user/200506.mbox/%3CDDEB64C8619AC64DBC074208B046611C7692DA#kronos.neoworks.co.uk%3E
We're running NServiceBus for a web application to handle situations where the user do "batch like" actions. Like fire a command that affects 1000 entities..
It works well, but during moderate load we get some deadlocks, this isn't a problem, just retry the message.. right? :)
The problem occurs when the next message arrives and tries to open a connection. The connection is then "corrupt".
We get the following error:
System.Data.SqlClient.SqlException (0x80131904): New request is not allowed to start because it should come with valid transaction descriptor
I've searched the web and I think our problem is a reported NH "bug":
A workaround should be to disable connection pooling. But I don't like that, since performce will degrade..
We're running NServiceBus 2.6, NHibernate 3.3.
Does anyone have any experience with this? Can a upgrade of NServiceBus help?
I’ve seen this in the past, if your design warrants, try breaking the transaction into two, if you flow the message transaction all the way to your database operations, any failures will have a cascading effect and it will impact (ideally it shouldn’t) any subsequent messages as well.
Instead of updating the 1000 entities in the command could you publishing an event to say that the command has been completed and then have several subscribers acting on this event to update effect entities. It sounds to me that a command that updates a 1000 entities should be split into a number of smaller commands. Take a look a the sagas to see how you can handle long running business process. For example, you might have something like, process started, step 1 completed, step 2 completed , process completed etc...
I have a cloudbees enterprise instance that I use for performance and automated UI testing.
The free instance (which is limited in memory) cannot support the memory or request per second that we have for testing.
I would like to have the instance automatically hibernated when I am not using it but have it wake up when requests come in. I would configure a jenkins job to wake the app up (by issuing a request) before kicking off my sauce lab based selenium jobs.
My question is how do I configure automatic hibernation? The control panel has minimum of one instance which I guess means that the one instance stays up.
You are right - currently automatic hibernation is only for free applications. When an application is hibernated (vs stopped) then it will be automatically woken whenever someone needs to access it.
What you could do for this is to have a job set your application to hibernated, say once a day, (or at certain time of the day when you know it won't be needed). When it is needed again - you won't need to do anything - simply accessing it will cause it to be activated (woken) again - so your test script can just insure that is the case (and ideally, after a test run, set it to hibernated again).
It really depends how often the app is needed - if you can work out what points it isn't needed and trigger the hibernate off that (eg after a test run) then that is ideal (you minimise cost).
Quick overview of our topology:
Web sites sending commands to an nServiceBus server, which accepts the commands and then publishes the correct pub/sub events. This service also has message handlers that can do some process against the DB in response to the command, for instance:
1 user registers on web site
2 web site sends nServicebus command to nServicebus service on another server.
3 nServicebus server has a handler for that specific type of command, which logs something to the database and sends a welcome email
Since instituting this architecture we started to get deadlocks on the DB. I have traced it down to MSDTC on the database server. If I turn that service OFF on the database server nServicebus starts throwing up errors, which to me shows that nServiceBus has been enlisting the DB update in the transaction.
I don't wish this to happen, I want to handle the DB failing myself, I only want the transaction to ensure the message is delivered to my nServicebus proxy service. I don't want a transaction from the web all the way through 2 servers to the DB and back.
Any suggestions?
EDIT: this post provides some clues, however I'm not entirely sure it's the proper way to proceed.. NServiceBus - Problem with using TransactionScopeOption.Suppress in message handler
EDIT2: The reason that we want the DB work outside the scope of the transaction is that the intent is to 'asynchronously' process these commands on another server so as not to slow down the web site and/or cause users to wait for these long running aggregation commands. If the DB is within the scope of the transaction, is that blocking execution on the website at the point where the original command is fired to the distributor? Is there a better nServicebus architecture for this scenario? We want the command to fire quickly and return control to the web site so the user can quickly proceed and not have to wait for our longish running DB command, which is updating aggregate counts and sending emails etc.
I wouldn't recommend having the DB work outside the context of the NServiceBus transaction. Instead, try reducing the isolation level of the transactions. This can be done by calling:
.IsolationLevel(System.Transactions.IsolationLevel.ReadCommited)
in the fluent configuration. You'll have to put this after .MsmqTransport() in v2.6. In v3.0 you can put this call almost anywhere.
RESPONSE TO EDIT2:
Just using NServiceBus will achieve your objective of not slowing down the website, regardless of the level of the transactions run on the other server. The use of transactions is to provide a guarantee that messages won't be lost in case of failure and also that you won't have to write your own deduplication logic.