Env:
Rails 5.0.5
Redis server v=4.0.1
There is nothing special on the server side. The problem is that a user have many pings for the same message (duplicates?).
Redis.new(url: 'redis://:auth#ip:port/db_number').pubsub('channels', 'action_cable/*') doesn't show extra connections.
Where can be the problem? Redis? Or something wrong with the settings of the app?
I restarted Redis server and the problem went away (it wasn't easy to do before cause the problem existed only in the production server).
Related
I've come into an issue post install of Rabbit MQ where it was all set up and configured with the web apps on the machine and communicating to local applications however the machine had to be moved to a different tranch of machines and renamed as a result. Now Rabbit MQ can no longer serve or handle comms as intended as it's config points to rabbit#PREVIOUS_MACHINE instead of rabbit#CURRENT_MACHINE.
In the rabbit MQ config however, to complicate this, there was some configuration that was done from the users on the system that were fed into the local apps that are then encrypted into that local app's database and used for communicating with all the local apps. the issue here is if I drop and recreate Rabbit MQ a make a new user this won't align to what the other internal apps are using and I believe they are not configurable post install so a reinstall of everything is the potential impact.
the question is, is it possible to re-config or update the current RabbitMQ installation files to now point at the local machine name instead of the previous machine name AND I guess by proxy is this something that would even work. The docs over at rabbitmq don't quite deal with this specific scenario, unfortunately from what I've read through.
so i want to confirm that RMQ is the absolute dogs tits of a black magic box.
anyway,
following these steps from here minus the first two
How to change RabbitMQ node name without changing my hostname
this is the inverse of my problem pretty much. But for those in the future who have this issue;
I had Rabbit MQ on another machine installed and running, the machines name was changed and the solution was to uninstall the service, delete the db and reinstall the service. SOMEHOW rmq manages to keep the config knowledge of all the queues that were in the db in the system and when you reinstall the service it brings all the queues back as well. the only issue I had after that was to remember my username and password that were not default user set ups and I did so that solved my issue. still have no idea how RMQ manages to remember the previous configs despite deleting the local db, crazy cool. very grateful to whoever built that into the tool
I have RabbitMQ Server 3.6.0 installed on Windows (I know it's time to upgrade, I've already done that on the other server node).
Heartbeats are enabled on both server and client side (heartbeat interval 60s).
I have had a resource alarm (RAM limit), and after that I have observed the raise of amount of TCP connections to RMQ Server.
At the moment there're 18000 connections while normal amount is 6000.
Via management plugin I can see there is a lot of connections with 0 channels, while our "normal" connection have at least 1 channel.
And even RMQ Server restart won't help: all connections would re-establish.
1. Does that mean all of them are really alive?
Similar issue was described here https://github.com/rabbitmq/rabbitmq-server/issues/384, but as I can see it was fixed exactly in v3.6.0.
2. Do I understand right that before RMQ Server v3.6.0 the behavior after resource alarm was like that: several TCP connections could hang on server side per 1 real client autorecovery connection?
Maybe important: we have haProxy between the server and the clients.
3. Could haProxy be an explanation for this extra connections? Maybe it prevents client from receiving a signal the connection was closed due to resource alarm?
Are all of them alive?
Only you can answer this, but I would ask - how is it that you are ending up with many thousands of connections? Really, you should only create one connection per logical process. So if you really have 6,000 logical processes connecting to the server, that might be a reason for that many connections, but in my opinion, you're well beyond reasonable design limits even in that case.
To check, see how many connections decrease when you kill one of your logical processes.
Do I understand right that before RMQ Server v3.6.0 the behavior after resource alarm was like that: several TCP connections could hang on server side per 1 real client autorecovery connection?
As far as I can tell, yes. It looks like the developer in this case ran across a common problem in sockets, and that is the detection of dropped connections. If I had a dollar for every time someone misunderstood how TCP works, I'd have more money than Bezos. So, what they found is that someone made some bad assumptions, when actually read or write is required to detect a dead socket, and the developer wrote code to (attempt) to handle it properly. It is important to note that this does not look like a very comprehensive fix, so if the conceptual design problem had been introduced to another part of the code, then this bug might still be around in some form. Searching for bug reports might give you a more detailed answer, or asking someone on that support list.
Could haProxy be an explanation for this extra connections?
That depends. In theory, haProxy as is just a pass-through. For the connection to be recognized by the broker, it's got to go through a handshake, which is a deliberate process and cannot happen inadvertently. Closing a connection also requires a handshake, which is where haProxy might be the culprit. If haProxy thinks the connection is dead and drops it without that process, then it could be a contributing cause. But it is not in and of itself making these new connections.
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
I recommended that this user upgrade from Erlang 18, which has known TCP connection issues -
https://groups.google.com/d/msg/rabbitmq-users/R3700QdIVJs/taDYKI6bAgAJ
I've managed to reproduce the problem: in the end it was a bug in the way our client used RMQ connections.
It created 1 auto-recovery connection (that's all fine with that) and sometimes it created a separate simple connection for "temporary" purposes.
Step to reproduce my problem were:
Reach memory alarm in RabbitMQ (e.g. set up an easily reached RAM
limit and push a lot of big messages). Connections would be in state
"blocking".
Start sending message from our client with this new "temp" connection.
Ensure the connection is in state "blocked".
Without eliminating resource alarm, restart RabbitMQ node.
The "temp" connection itself was here! Despite the fact auto-recovery
was not enabled for it. And it continued sending heartbeats so the
server didn't close it.
We will fix the client to use one and the only connection always.
Plus we of course will upgrade Erlang.
Dears,
I am facing a problem with my weblogic 12c Server. I have 3 managed nodes and one admin server. I have configured an Oracle Datasource. Whenever any network glitches or database downtime happens, weblogic server also goes down completely. Is there any configuration which I can make to hold on the server until everything comes back.
Servers are trying to restart automatically.But it is in STARTING state. I have to again start all servers and it is taking time. Your suggestions will help me a lot. Thanks in advance.
If you don't want your servers to restart automatically, the server failure action can probably be tweaked:
{managed_server} -> Configuration -> Overload -> Failure Action
However, probably you should be more interested in exploring if you are missing connection timeout config for your datasource.
I have a Mule 3.6.1 ESB server up and running and a MMC server to manage it. The problem I'm seeing is every time I need to restart the ESB server manually (to fix something) the MMC shows the server with an error (red circle) even when the ESB is back up and running normally (APIs are responding without issues).
Any idea how to fix this problem ?
This issue may occur for various reasons, the console polls the managed instances every certain period so you need to at least wait for that period of time to see this, also if you have something in between the console and the managed instance such as a proxy it might be caching the responses, finally I would check if it is a display issue, by trying to reload the page.
I have a WCF application with a couple thousand clients connecting to a pair of services running under IIS. What I've noticed is that some of these clients get into a hung state, and I'm trying to reproduce this.
When this problem was first noticed, I had not modified the throttling configuration and the services were set to ConcurrencyMode.Single. One thing I noticed was that an IISReset on the server caused many clients to hang. Yet pulling this same stunt on the client running against IIS on my local machine doesn't seem to cause the problem.
I caught this only once in the wild, but didn't have debugging enabled at the time. The symptom I witnessed was that the client appeared to be trying to open a connection to the web server, but did not succeed. While monitoring with Fiddler, I saw no attempt to reach the service endpoint. Obviously that makes me suspect the client proxy.
I have a very solid hunch as to what's happening -- namely I've been using "Close()" instead of "Abort()" when the service throws an exception, which I believe is causing the channels to become corrupted. But considering the effort to get a new version out there, I need to reproduce this problem by causing a client on my own machine to hang before I can start making changes to the code.
Where should I start?
Thanks in advance,
roufamatic
Have you got any logging turned on? This could help in diagnosing the problem. It can be done completely in config, so no need to build a new version. Use the Service Configuration Editor tool to set it all up. The Visual Studio 2008 Training Kit has a good tutorial on how to use logging and the log viewer.
I suppose this was too vague a question though I was mostly curious what people might suggest. As it turns out there was a nontrivial difference between my workstation and a production environment that, once resolved, allowed me to see the problem. In this case, somehow using Fiddler to watch the traffic actually prevented the error from occurring! Now to ask another question.