We have been migrating to .net core console app microservices. currently, each microservice works in a chain and puts messages in rabbitmq, then the next service picks a message from rabbitmq, processes it, then puts in another rabbitmq....we have around 9 services.
We are seeing issues where services fail and have no idea why, but often see problems with rabbitmq connections or network issues hitting the next server (some vm's have all services hosted on the same box, others are distributed between boxes)
I've been looking at envoy proxy as it deals with the circuit breaker etc stuff and claims to have observability
However, I cannot find anywhere online that has anyone using envoy proxy with rabbitmq
Can envoy proxy be used with rabbitmq in this manner?
Or does envoy proxy act as the queue?
We deal with about 4,000 messages a sec currently, and we need to process in near as real-time as possible.
Envoy does not act as the queue, so it can't replace your message-based communication system. It can, however, proxy traffic to/from the RabbitMQ servers to give you some bits of what you're looking for.
What you'd do is use the TCP Proxy capability to setup TCP reverse proxies to RabbitMQ. Then your servers should all connect to the Envoy proxy rather than directly to the message queue. Envoy's built in stats will then output metrics on the TCP connections (all the RabbitMQ protocols seem to be TCP) that it handles. It also does intrinsically support circuit breakers, timeouts, retries, etc, so you'll get all those. But you'll definitely have to tweak those to your particular deployment.
We've done this pattern multiple times at my company, just with Kafka rather than RabbitMQ. However, since they're both TCP based it should work similarly.
Related
I need your help to suggest me how best I can achieve load balancing using the below diagram. here I am trying to create 2 machines with Master and expecting that the consumer/publisher application will use one common URL( a load-balanced one), where I should not expose the individual VM machine info and port ID. just that load balancer should take care of routing..
this is typically what we do with help of F5 load balancer or HTTP load balancer ..just wondering can be achieved over ActiveMQ and its advisable..?
on other side, I also tried configuring this way on weblogic to consume data from ActiveMQ queue
failover://(tcp://localhost:61616,tcp://localhost:61617)?randomize=true but this does not help.. or WebLogic is not understanding this format.
Messaging connections are stateful. They are not stateless like HTTP connections, and therefore cannot be load-balanced in the same way as HTTP connections. It may be possible to configure an F5 to deal with stateful messaging connections, but I can't say for sure. I'm not an expert on F5.
Both the ActiveMQ Artemis broker itself as well as the JMS client shipped with the broker have load-balancing functionality built in. There's too much to cover here so I recommend you review the clustering documentation for the relevant details.
You might also try using the broker balancer feature. It's currently experimental, but it should be ready to use in the 2.21.0 release coming in the March/April time-frame. It can act like an F5 for your messaging connections, but it can do some more intelligent things like always sending certain clients to the same node which can facilitate certain use-cases which are not possible in a traditional cluster.
The URL failover://(tcp://localhost:61616,tcp://localhost:61617)?randomize=true which are you using is for the OpenWire JMS client shipped with ActiveMQ 5.x. If you're using the core JMS client shipped with ActiveMQ Artemis then you should be using a URL like this instead:
(tcp://localhost:61616,tcp://localhost:61617)?ha=true
I am looking in for the best way to implement the RabbitMQ consumer by using .Net Client which should be run as windows service.
I referred the RabbitMQ documentation and found the way to consume messages by using .Net client (https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html).
My current scenario is like, RabbitMQ is installed in AWS VM machine. I have to install dotnet client consumer service resides in On-premise network which should consume messages.
Which one is the best way, to always listen the Queue (AMQP protocol) or HTTP API which should get messages on demand (https://pulse.mozilla.org/api/).
Please advise.
Thanks,
Vinoth
I believe the answer is "neither." You should have your message queue as a back-end service behind the firewall, and expose your application functionality through a set of carefully-specified web services. The web services, which are exposed through the firewall but can communicate to services behind the firewall, would produce messages that would be transmitted to the server. Any services needing to produce or consume messages would need to do so via the web services, which would perform safety/security checking prior to forwarding the request on to the AMQP server.
If you need to expose AMQP directly to clients (i.e. that is the purpose of your app), then the recommendation is to do so via STOMP. I think a valid use case for exposing AMQP directly over the internet would be a rare thing to come across. The security implications of doing so would be immense.
I am designing a system where a huge number of real-time data generated from devices is to be transferred to subscribers preferably over websockets. I have decided to use Spring STOMP Websockets as it was quicker to set-up, understand and had a few things supported out of the box like RabbitMQ and Security. And also because the plan is to use Spring for another REST API so Spring as a choice of tech stack. RabbitMQ is the message broker that I have decided on. However I can not find good amount of guidance on how to scale such a system.
The possible solution I am thinking of is:
To add HAProxy in front of STOMP broker instances and also between
STOMP Brokers and a RabbitMQ cluster, HAProxy will act as a
load-balancer in both cases. Spring STOMP broker will then be pointing to the HAProxy as broker relay host. The requirement is to have high availability and no data loss.
As I do not have prior experience with Websockets, I would like to get guidance on if this solution sounds correct or if there is anything that I am missing here?
Note: In this system, both the message producers and consumers are actually websocket Java clients. I took the sample code from https://github.com/nickebbutt/stomp-websockets-java-client and created two separate clients - One that only sends the messages i.e. device data(Producers) and other that subscribes to these messages(Consumer). Thus both connect using same websocket URL to same STOMP broker. With above system implementation the clients will point to HAProxy for websocket connection.
Just an updated on this, I did experimentation by creating the above set-up and it worked i.e. I was able to connect to websocket stomp server/send/receive data with RabbitMQ broker and use of HAProxy load balancing as described. The broker host/port configured in Spring was pointing to HAProxy which in turn was forwarding requests to RabbitMQ backend. Similarly, the websocket clients were connecting to Spring STOMP websocket server application via HAProxy.
We usually use message passing to send messages to decoupled services. This makes service discovery a non-issue, because (with AMQP in RabbitMQ for instance) you can use the broker's routing capability to dispatch messages to the right queues that feed the correct services. Load balancing is also handled by the message broker.
Enter kubernetes.
The use case that is usually laid out when talking about service replication and re-spawning failing services, is when your clients use some active protocol like http to contact a service, even if this service handles requests asynchronously. In this context, it is a natural fit to have replication controllers, that manage a group of services and a single entry point to load balance between them.
I like kubernetes' intuitive concepts, like rolling deployments, but how to you control this beasts that don't have an http interface ?
UPDATE:
I am not trying to set up a cluster of message brokers. I am looking at message consumers as services. Service clients don't connect directly to the services, they send messages to the message broker. The message broker acts as a load balancer in a way, and dispatches the messages to the subscribed queue consumers. These consumers implement the service.
My question gravitates around the fact that most usage patterns in demos handle services that are called via http, and kubernetes does a good job here to create a service proxy for these services, and a replication controller. Is it possible to create replication controllers for my kind of service, which does not have a http interface per se, and have all the benefits of rolling updates, and minimum instances?
I'm not sure I entirely understand the question. Are you asking how to use RabbitMQ with Kubernetes? Or how to set up a RabbitMQ cluster: https://www.rabbitmq.com/clustering.html? Or how rolling updates interact with RabbitMQ? Or something else?
I think you should be able to create one service and one replication controller per server, and then use the service DNS names in the cluster configuration file. This is the current approach used to run Zookeeper, also. We have a long-standing TODO to make this less verbose (https://github.com/GoogleCloudPlatform/kubernetes/issues/260), but the current approach should be straightforward. You do lose the ability to use a single kubectl rolling-update command to update the cluster, but it's also straightforward to update the instances individually.
I'm trying to figure out what is the best solution to work with rabbitmq cluster via wcf.
Current setup:
2 IIS web servers (act as message produces and post messages to queue via amqp wcf client).
2 servers with rabbitmq broker (clustered with mirrored queue, rabbit1 and rabbit2)
Windows service ( worker) with hosted amqp wcf service that listens to incoming messages.
Web role posts messages to rabbit1 node and worker listens to rabbit1 node too. If rabbit1 node fails system(both web and worker) should switch to rabbit2. And that's the question, how to implement this in more elegant way rather than handling connection failures in application code.
First and the only approach I see now is to use wcf4 routing backup endpoints feature. This way solves problem on client side(web role) only but doesn't solve problem on wcf service side(worker role).
One way is to create a wrapper around your service host, used for storing a list of connection strings (which can come from config).
Add a handler to the service faulted event, where you can close and reopen the host with a different connection string.