Can I configure Glassfish to drop any request that takes longer than 10 seconds to process?
Example:
I'm using Glassfish to host my web service. The thread pool is configured to have max 5 connections.
My service has a method that does this:
System.out.println("New request");
Thread.sleep(1000*1000);
I'm creating 5 requests to the service and I see 5 messages "New request" in the log. Then the server stop to respond for a looong time.
In live environment all requests must be processed in less than a second. If it takes more time to process then there is a problem with the request and I want Glassfish to drop such requests but stay alive and serve other requests.
Currently I'm using a workaround in the code. At the beginning of my web method I launch a separate thread for request processing with a timeout as it was suggested here: How to timeout a thread
I do not like this solution and still believe that there must be a configuration setting in the Glassfish to apply this logic to all requests, not to just one method.
Related
I have a WCF SOAP service that receives too many synchronous requests from other systems
I am having problem when too many request comes at that time IIS Queue will not provide proper result and server memory and CPU usage is gone high and discard request or time out
I did a normal Load test (100 requests with 100 concurrent user) and the IIS started to discard the requests after the maximum queue length reach as well as all the request coming delays to provide the response and Other requests coming in are delayed until the first one either times out, or responds.
Below is server configuration
WCF application code is tested with Resharper tool and there is no object or memory dispose issue
Is there any settings for setup application pool or worker process to manage queue ?
Can i apply web garden in Application ?
Please help me to solve this issue
Thanks in Advance
We have an application pool dedicated to a WCF service that is called infrequently (maybe 15-20 times per day). The calls can take several minutes, however, and the other day we got burned when IIS recycled the app pool while the call was still processing because the shutdown timeout ran out.
We're considering using request limit recycling, instead, but my question is this: When the application pool recycles "after x requests", is that after the xth request completes? Or does it kick off the request, start the overlapped process to handle new requests, then subject the xth request to the same shutdown timeout that currently burns us?
Question in a similar vein:
How to detect if the current application pool is winding up in IIS7.5 and Asp.Net 3.5+
Check your Shutdown Time Limit setting on the app pool.
Regardless of how you do the recycling, this setting is checked to determine how long a request is allowed to carry on for before being forcibly shut down.
When an app pool is recycled, IIS attempts to drain the running requests from the app pool first, and then a in the meantime a new app pool is already started which accepts new requests. By making the setting high enough to accommodate your long running requests, you will allow IIS to safely drain the old app pool.
I recommend you do the following.
1- Create a bool Ping() { return true;} method under your WCF service.
2- Create an IIS web application responsible of polling the Ping() method. This is the only way i found to keep my WCF services alive.
3- WCF long running operations must be called also from another background IIS process (web app) that must read from a message queue and call the WCF operation. So you need to log the WCF long running operation call requests in queues. This way, you will have the possibility of retrying the call if the app pool where your WCF services are hosted shuts down.
Using Weblogic 12c, i have a big problem: when calling a rest service using differents clients (java client or using command line like curl) there is no problem.
BUT when the client is in C#, the request is executed severals time (each minutes) by one different thread of the pool so it implies lot of errors in log files because:
The C# client get its response and close the connection, but the others 'duplicated' requests (by threads pools) creates stacks errors because the service can not write a response (there is no client to get the response).
Some precisions:
- There is no Stuck thread in my case.
- Using Tomcat there is no problem
- There is the same problem using new installed Weblogic12c Server (so there is no custom configuration)
- The http headers are the same between C# or others clients
- The same data tests are used to reproduce the problem
check access.log to check how many requests are received by the server, if its more in case of c# then its a problem there not the server.
You can also enable http debug to get more details about the incoming request.
the service endpoint socket connection always abort 10 mins after last use.
the above image shows that the last use of the end point was 10:18:21. after 10 mins, activity Aborted 'System.ServiceModel.Channels.ServiceChannel' happened. is the 10 mins time out a default setting for WCF endpoint socket connection? can I set the timeout to be infinite? notice the abortion happens on a separate thread(thread 16).
or did I not configure the endpoint correct on the service endpoint?
The socket connection timeouts needs to be configured on both client and server side (smaller of the two will prevail). These should be done via binding configuration (in config or code). The timeouts can be done via inactivityTimeout setting of a reliableSession, in combination with recieveTimeout of the netTcpBinding. You can also consider using idleTimeouts in the connection pool settings option of netTcpBinding. It is typically not recommended to set infinite timeouts unless you have very specific needs that need to be met and have service usage parameters that safegaurd against infinite timeouts.
We observed the following behavior on one of the servers hosting a WCF service on IIS 6.0:
The IIS log shows a high value for time-taken (> 100000)
The HTTP status code is 200
sc-win32-status code shows a value of 64
I found out that sc-win32-status code of 64 indicates "The specified network is no longer available"
Initially I suspected that it could be because of limits set on MinFileBytesPerSecond, which sets the minimum throughput rate that HTTP.sys enforces when sending data from the client to the server, and back from the server to the client.
But the value for sc-bytes and cs-bytes indicate that the amount of data is sent is within the range generally observed for the service.
Also note that the WCF service is hosted on four boxes and is load-balanced, but the problem occurs only one of the servers. (but not essentially on the same server). The problem is also intermittent.
Has anybody else encountered this error? Any clues about what could be wrong?
Update
Note: Observation on IIS 7.5 (IIS version does not really matter)
I was able to replicate the issue. The issue occurs if:
1. The WCF service takes a long time to respond
2. The client proxy times out before it receives a response from the server. In this case it leads to TimeoutException on the client.
3. The server keeps waiting for TCP ACK for the client, which it would never receive.
Hence a long timeout (TCP socket timeout (default value: 4 minutes) and sc-win32-status of 64
So essentially it appears that WCF code is taking a long time to respond and the client is timing out, what I observe in IIS log is just a symptom and not a problem.
The behavior you are describing will also occur if you exceed a WCF service's max sessions, calls or instances (depending on how you have your service instancecontext mode configured). If you observe the System.ServiceModel performance counters for %max concurrent sessions and/or %max concurrent calls (again depending on your service's instance context), you may see a correlation with the IIS log entries.
Note that these maxes can be configured in the service throttling behavior.
https://msdn.microsoft.com/en-us/library/vstudio/system.servicemodel.description.servicethrottlingbehavior(v=vs.100).aspx
I saw your question again and wanted to point out that I found a solution for this. It turned out to be this piece of code in the web.config:
<pages smartNavigation="true">
After turning this off I stopped receiving the same time-out errors. See also the answer here
IIS put the services into sleep to save recources.
Copied from here (WCF REST Service goes to sleep after inactivity)
The application pool hosting your service defines Idle Time-out property (advanced settings of app pool in IIS management console) which defaults to 20 minutes. If no request is received by the app pool within idle timeout the worker processes serving the pool is terminated. After receiving a new request the IIS must start the process again, the process must load application domain and all related assemblies, compile .svc file, run the service host and process the request.The solution can be increasing idle time-out but the meaning of this time-out is correct handling of server resources. If the process is not needed it should be stopped. Another ugly workaround is using some ping process (for example cron job or scheduled task on the server) which will regularly ping call some method on the service or page in the same application.