Redis StackExchange Client - Frequently receiving "Timeout exceptions", "Redis connection exception", "No connection available to service" - redis

I am frequently getting the below mentioned errors, the dll version used in the project is - 1.0.488.0
System.TimeoutException: Timeout performing GET
StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: GET
No connection is available to service this operation: EXISTS
Can anyone help me out in figuring what the issue can be?
Have also created an issue on StackExchange's Github repo for the same
Issue created on Github for the same

It looks like your connection broke. And when it did, any commands already sent to Redis would have timed out on the client application, even though they could have executed on the server. If you upgrade to a later version of the StackExchange.Redis client, you will get richer diagnostics information about what the state of the threadpool, CPU etc was on the client application side.

Related

Google Cloud SQL Second generation -> "Aborted connection"

I am running a Java + JPA/Hibernate application on Appengine and switched my database from the first generation Google Cloud SQL instance to the second generation and now get a lot of this errors:
2017-05-20T22:49:53.533247Z 2235 [Note] Aborted connection 2235 to db:
'mydb' user: 'root' host: 'cloudsqlproxy~myip'
(Got an error reading communication packets)
As far as I can tell, most of these error occur during database requests inside task queue tasks.
This did not happen with the first generation. How can this be avoided?
Τhe "Aborted connection nnnn to db:" message is triggered when an existing connection is terminated improperly as described by Google’s documentation. Most of the aborted connections happen because of the termination of your connection was not correct or because of networking problem between the server and the client, as described in the documentation here.
I advise you to follow the Google’s documentation about managing Cloud SQL connections, emphasizing on the “connection pools” section and of course the “opening and closing connection” section.
Managing database connections talks about "close connection properly". However, in my case, the error still occurs when I use GCP cloud function to connect GCP cloudSQL.
A Google group says that, unless you use NullPool OR dispose the engine explicitly, the error message will always occur. Also, it does not suggest to use engine.dispose().
So I wonder what is the best way to release the resource of connection pool without generating error message on CloudSQL?

Database Sync failing after a day

We setup database sync between two databases on the same server. It worked fine yesterday and then stopped working today. I tried killing connections to the database and stopping the web apps that are connected to the database thinking maybe it was a connection limit. I also reset the user and pass after verifying the connection is correct.
This is the error we're getting:
Database re-provisioning failed with the exception "The current operation could not be completed because the database is not provisioned for sync or you not have permissions to the sync configuration tables." For more information, provide tracing ID ‘b4b76a8c-38ae-4b48-ad08-6c07933c23c1’ to customer support.
The error log indicated that the previously provision related operation was not completed yet, so you were not able to re-provision the sync group at the same time.
May I know whether you are still experiencing this problem? If yet could you please provide the latest error log? I'll update the answer then.

Do you need to open client firewall for WS-AT to work with WCF?

I am getting the following error trying to communicate from WCF -> WCF across the internet with a SSL certificate. I have 'No authentication required' checked in the WS-AT configuration on the server.
After a few seconds I'm getting the following message on my client, and trying to figure out what is wrong in my configuration (or understanding of WS-AT).
The flowed transaction could not be
unmarshaled. The following exception
occurred: The WS-AtomicTransaction
protocol service could not unmarshal
the flowed transaction. The following
exception occured: A fault reply with
code CoordinatorRegistrationFailed was
received. The fault reason follows:
The WS-AT protocol service failed to
register with its coordinator. A
connection could not be established.
Looking at the log file on the server I find a link to http://msdn.microsoft.com/en-us/library/aa702582(VS.90).aspx and the following trace message
Microsoft.Transactions.TransactionBridge.RegistrationCoordinatorFailed
Traced if the local TransactionManager is not able to
Register with its superior
TransactionManager due to the
inability to send a message.
So I'm wondering... is the server attempting to communicate with the client through a different channel to the SSL connection originally established by the client. I'm pretty sure thats what it's trying to do for me to end up with this message. The client is my laptop (currently in Starbucks) so there's no chance of any incoming connections.
So is it possible to use WS-AT transactions where only the server has an open firewall? Assuming it is possible - what do i have to do to allow for it? I'm finding very little information about necessary configuration - even in my massive Programming WCF book .
Bonus: Here's a few small tips for anyone trying to get WS-AT working :
If when trying to install the WS-AT tab you can't get it to appear - and you're running an x64 operating system - make sure you're using the x64 version of RegAsm.exe:
C:\Windows\Microsoft.NET\Framework64\v2.0.50727\RegAsm.exe /codebase wsatui.dll
If once installing WS-AT, configuring it for incoming connections and restarting MCAT you still get errors about the protocol being disabled you may have forgotton to restart IIS(!)

MQ With WLS Foreign Server

I am facing two issues when i try to connect to MQ which is deployed on a Remote Server from Weblogic Server(WLS) by creating a Foreign Server.
1. When I try to connect to MQ Queuemanager in Bindings mode(after importing the .Bindings file) i keep getting the below error in WLS console:
java.lang.UnsatisfiedLinkError: no mqjbnd05 in java.library.path
If i Switch the Transport to Client i keep getting:
JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name 'localhost'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
Has anyone seen this, and are there any performance implications which dictate the use of client over bindings and vice versa?
TIA
Finally i was able to resolve this, i had to recreate the .bindings file in the client mode, with changes to the IVTsetup.bat which is most likely present in
C:\Program Files\IBM\WebSphere MQ\java\bin, I had to run this
def qcf(psQCF) TRANSPORT(CLIENT) HOST(SMEKA) PORT(1415) CHANNEL(ps_SRV_CHANNEL) QMGR(psQM)
to generate the .bindings file.
Refer to this link for more details:
http://publib.boulder.ibm.com/infocenter/wbihelp/v6rxmx/index.jsp?topic=/com.ibm.wbia_adapters.doc/doc/peoplesoft/peopleso103.htm
Where the question states that I try to connect to MQ which is deployed on a Remote Server from Weblogic Server I assume this means that WLS and WMQ are on two different hosts. If that is the case, then a bindings mode connection (which relies on shared memory segments) won't work.
The client mode connection appears to be using a CF that is pointed to localhost rather than the IP or hostname of the WMQ server. This would work for an application on the same host as the queue manager but not when the app and QMgr are on separate servers.
As far as choosing between client and bindings mode, the answer is that if the QMgr is local use bindings. This provides highest reliability, best performance and XA transactionality. When using client mode, two-phase XA commit is not supported without the Extended Transactional Client. Per the JMS specification, there is an ambiguity that can exist if an app loses the connection during a COMMIT call. Depending on how the app handles this it's possible to end up with duplicate messages. (The JMS spec refers to these as "functionally duplicate.") This ambiguity is much less likely to occur with a bindings mode connection since there is no network latency and not even any traversal of the IP stack or interface. So use bindings mode where possible.
UPDATE:
Removed note about Extended Transactional Client being a chargeable component. As of April 24th, XTC is free of charge for all versions of WMQ on all platforms.

WCF Service hangs and clients receive a ServiceModel.CommunicationException

My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin