my application (Node.js) is using moleculer for microservices and redis as transporter. However, I find that the application will have this log Redis-pub client is disconnected every 10 minutes, then reconnect with the log Redis-pub client is connected after a few seconds. This is a problem because if a client send a moleculer action during this time, it will fail.
Any idea what is causing this? Let me know if more information is needed.
Azure Cache for Redis currently has a 10-minute idle timeout for connections, so the idle timeout setting in your client application should be less than 10 minutes. Most common client libraries have a configuration setting that allows client libraries to send Redis PING commands to a Redis server automatically and periodically. However, when using client libraries without this type of setting, customer applications themselves are responsible for keeping the connection alive.
More info: https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-best-practices-connection#idle-timeout
Related
Dropbox has more than 300M user.Dropbox desktop application need to keep connection alive with dropbox server for every updates.
But how does dropbox server keep connection alive with all its desktop user?
The dropbox client keeps a TCP connection constantly open to listen for server-side notifications. When it receives a notification, the client initiates an HTTPS conversation to see what changed and download it. When something changes on the client side, it also initiates an HTTPS conversation to update the files on the server.
Source: http://www-net.cs.umass.edu/imc2012/papers/p481.pdf
The Dropbox client keeps continuously opened a TCP
connection to a notification server (notifyX.dropbox.com),
used for receiving information about changes performed else-
where. In contrast to other traffic, notification connections
are not encrypted. Delayed HTTP responses are used to implement a push mechanism: a notification request is sent by the local client asking for eventual changes; the server response is received periodically about 60 seconds later in case of no change; after receiving it, the client immediately
sends a new request. Changes on the central storage are instead advertised as soon as they are performed.
While the decrypted headers give no indication of what servers Dropbox uses to keep so many open TCP connections, people report being able to keep over 600k (https://stackoverflow.com/a/9676852/15472) or even over 1M (http://blog.whatsapp.com/196/1-million-is-so-2011). With enough load-balancing, 300M users, of which only a fraction of which are connected simultaneously and actively share data within each other, certainly seems within reach.
I doubt that all 300M users are connected at the same time... And by the amount of storage they provide, they will have enough servers to handle the needed amount of connections, maybe 1% of their user count at a time.
If you like to investigate yourself, you could use tools like TCPView (part of Sysinternals Suite) to check which connections are opened by the application, or Wireshark to check the transferred data.
I assume that you mean 'update' of storage content; that could also happen on fixed intervals by opening a connection, getting the files list and closing the connection afterwards. In this case the connection would be used for a few seconds in an interval of e.g. 5 minutes. This would again reduce the number of needed simultaneous connections by factor ~100.
I have added the serverSessionTimeout=3 in the worklight properties file, I have edited this in the WAS server and restarted the application, but when i test the app and keep it idle for 3 mins, i cant see the server session time out automatically, is there anything we need to call on the mobile client to check if the server session time out is done or not. please help
Thanks
djrecker
Please look at the following documentation to get a better understanding of how the serverSessionTimeout and heartbeat property effect eachother as well as other tuning mechanisms:
http://pic.dhe.ibm.com/infocenter/wrklight/v6r0m0/index.jsp?topic=%2Fcom.ibm.worklight.help.doc%2Fadmin%2Fc_optimizing_and_tuning_of_ibm_wl_server.html
Specifically the "Worklight Server internal configuration" section:
"Consider the following factors:
A session is an object that is stored in server memory for each connecting device. Among other things, it stores authentication information. The number of active sessions is the difference between the number of opened sessions and the number of sessions that are timing out because of inactivity. The serverSessionTimeout property configures the session timeout and affects the server memory consumption. The default session timeout is 10 minutes.
The mobile client "heartbeat" property causes the mobile client to ping the server while the app is in the foreground. This feature prevents the server session from timing out.
When a mobile app runs in the background, it no longer interacts with the server or sends a “heartbeat”. The server session drops after the specified server session timeout period.
For example, suppose every minute 1,000 users start a session against the server. Even if they exit the application after 3 minutes, their sessions remain active on the server for 10 minutes, leaving 10 x 1,000 = 10,000 active sessions.
"
You could try disabling the heartbeat using
WL.Client.setHeartBeatInterval(-1)
Maybe during the initialization of your app
Documentation at:
http://pic.dhe.ibm.com/infocenter/wrklight/v6r0m0/index.jsp?topic=%2Fcom.ibm.worklight.help.doc%2Fapiref%2Fr_wl_client_setheartbeatinterval.html
With that the session timeout may work with the app in foreground too.
If the app is in the foreground, a heartbeat message will be sent
to the server and the session will not time out.
So the app should be placed in the background for the timeout to occur.
If that does not help, please try the test for a period of 10 minutes,
as this is the default session timeout value.
Normally the value of the serverSessionTimeout is greater than the
heartbeat interval so that the session is kept alive when idle and
as long as the application is in the foreground.
So as long as the app is in the foregraound,
by design it will never invalidate the session.
But on the other hand, if it is backgrounded then there are no heartbeats
and then past the default 10 min in the background state, the session
is invalidated. This is the intention here.
I've noticed that when I run a php script on my redis server (simple set / get) that it will load in under 1 ms. If I have two servers, a web server and a redis server, it will take a good 15 ms for the web server to connect, set, and get. Is there a way to make a constant connection between the two servers so I don't need to reconnect every single time the script is called?
It depends on the client library you are using to communicate with redis if it supports/creates a persistent connection or a pool of pre-created connections in order to save initial handshaking for each request.
We observed the following behavior on one of the servers hosting a WCF service on IIS 6.0:
The IIS log shows a high value for time-taken (> 100000)
The HTTP status code is 200
sc-win32-status code shows a value of 64
I found out that sc-win32-status code of 64 indicates "The specified network is no longer available"
Initially I suspected that it could be because of limits set on MinFileBytesPerSecond, which sets the minimum throughput rate that HTTP.sys enforces when sending data from the client to the server, and back from the server to the client.
But the value for sc-bytes and cs-bytes indicate that the amount of data is sent is within the range generally observed for the service.
Also note that the WCF service is hosted on four boxes and is load-balanced, but the problem occurs only one of the servers. (but not essentially on the same server). The problem is also intermittent.
Has anybody else encountered this error? Any clues about what could be wrong?
Update
Note: Observation on IIS 7.5 (IIS version does not really matter)
I was able to replicate the issue. The issue occurs if:
1. The WCF service takes a long time to respond
2. The client proxy times out before it receives a response from the server. In this case it leads to TimeoutException on the client.
3. The server keeps waiting for TCP ACK for the client, which it would never receive.
Hence a long timeout (TCP socket timeout (default value: 4 minutes) and sc-win32-status of 64
So essentially it appears that WCF code is taking a long time to respond and the client is timing out, what I observe in IIS log is just a symptom and not a problem.
The behavior you are describing will also occur if you exceed a WCF service's max sessions, calls or instances (depending on how you have your service instancecontext mode configured). If you observe the System.ServiceModel performance counters for %max concurrent sessions and/or %max concurrent calls (again depending on your service's instance context), you may see a correlation with the IIS log entries.
Note that these maxes can be configured in the service throttling behavior.
https://msdn.microsoft.com/en-us/library/vstudio/system.servicemodel.description.servicethrottlingbehavior(v=vs.100).aspx
I saw your question again and wanted to point out that I found a solution for this. It turned out to be this piece of code in the web.config:
<pages smartNavigation="true">
After turning this off I stopped receiving the same time-out errors. See also the answer here
IIS put the services into sleep to save recources.
Copied from here (WCF REST Service goes to sleep after inactivity)
The application pool hosting your service defines Idle Time-out property (advanced settings of app pool in IIS management console) which defaults to 20 minutes. If no request is received by the app pool within idle timeout the worker processes serving the pool is terminated. After receiving a new request the IIS must start the process again, the process must load application domain and all related assemblies, compile .svc file, run the service host and process the request.The solution can be increasing idle time-out but the meaning of this time-out is correct handling of server resources. If the process is not needed it should be stopped. Another ugly workaround is using some ping process (for example cron job or scheduled task on the server) which will regularly ping call some method on the service or page in the same application.
Can I configure Glassfish to drop any request that takes longer than 10 seconds to process?
Example:
I'm using Glassfish to host my web service. The thread pool is configured to have max 5 connections.
My service has a method that does this:
System.out.println("New request");
Thread.sleep(1000*1000);
I'm creating 5 requests to the service and I see 5 messages "New request" in the log. Then the server stop to respond for a looong time.
In live environment all requests must be processed in less than a second. If it takes more time to process then there is a problem with the request and I want Glassfish to drop such requests but stay alive and serve other requests.
Currently I'm using a workaround in the code. At the beginning of my web method I launch a separate thread for request processing with a timeout as it was suggested here: How to timeout a thread
I do not like this solution and still believe that there must be a configuration setting in the Glassfish to apply this logic to all requests, not to just one method.