i've a .NET WebApi project running in k8s as docker image.
Now i need to listen from a Service Bus and handle messages.
Is better to have a different microservice in a different container or can i listen from messages in webapi projects?
In high volume scenario, what is the best approce?
Thanks
Related
I am looking in for the best way to implement the RabbitMQ consumer by using .Net Client which should be run as windows service.
I referred the RabbitMQ documentation and found the way to consume messages by using .Net client (https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html).
My current scenario is like, RabbitMQ is installed in AWS VM machine. I have to install dotnet client consumer service resides in On-premise network which should consume messages.
Which one is the best way, to always listen the Queue (AMQP protocol) or HTTP API which should get messages on demand (https://pulse.mozilla.org/api/).
Please advise.
Thanks,
Vinoth
I believe the answer is "neither." You should have your message queue as a back-end service behind the firewall, and expose your application functionality through a set of carefully-specified web services. The web services, which are exposed through the firewall but can communicate to services behind the firewall, would produce messages that would be transmitted to the server. Any services needing to produce or consume messages would need to do so via the web services, which would perform safety/security checking prior to forwarding the request on to the AMQP server.
If you need to expose AMQP directly to clients (i.e. that is the purpose of your app), then the recommendation is to do so via STOMP. I think a valid use case for exposing AMQP directly over the internet would be a rare thing to come across. The security implications of doing so would be immense.
We are working towards an architecture like one below but we will have micro services on cloud and some on premises which will talk to each other using queue(s) and bus(es),
Now I am confused with where we should host MassTransit and RabbitMq, also should it be a ASP.NET Core project on its own ? if yes what I will be doing in it ? starting a bus ? creating queues ? I am not able to move forward with this
The simple MassTransit examples are just that, the absolute simplest examples of interacting with queues.
RabbitMQ is your message broker. It is hosted separately.
MassTransit is a development framework that makes it much easier to interact with RabbitMQ (or Azure Service Bus) by abstracting away the implementation-specific "plumbing."
You write any number of .NET services that either publish messages to a queue, or subscribe to queues.
We recently worked on something similar, the way we did it is:
RabbitMQ was hosted separately, and buses/queues creation and management were done from the services that use messaging.
For each service that receives messages you use Maastransit to create a queue because service will be receiving messages using this queue.
You will be using publish/subscribe way of messaging so as mentioned above, inside each service, create a queue with logical name and connect to RabbitMQ server address.
Services that represent senders will publish messages of a custom type you create, and services that represent receivers will subscribe to this type of messages by having a consumer for this type registered inside the bus created.
Hope it helps.
We usually use message passing to send messages to decoupled services. This makes service discovery a non-issue, because (with AMQP in RabbitMQ for instance) you can use the broker's routing capability to dispatch messages to the right queues that feed the correct services. Load balancing is also handled by the message broker.
Enter kubernetes.
The use case that is usually laid out when talking about service replication and re-spawning failing services, is when your clients use some active protocol like http to contact a service, even if this service handles requests asynchronously. In this context, it is a natural fit to have replication controllers, that manage a group of services and a single entry point to load balance between them.
I like kubernetes' intuitive concepts, like rolling deployments, but how to you control this beasts that don't have an http interface ?
UPDATE:
I am not trying to set up a cluster of message brokers. I am looking at message consumers as services. Service clients don't connect directly to the services, they send messages to the message broker. The message broker acts as a load balancer in a way, and dispatches the messages to the subscribed queue consumers. These consumers implement the service.
My question gravitates around the fact that most usage patterns in demos handle services that are called via http, and kubernetes does a good job here to create a service proxy for these services, and a replication controller. Is it possible to create replication controllers for my kind of service, which does not have a http interface per se, and have all the benefits of rolling updates, and minimum instances?
I'm not sure I entirely understand the question. Are you asking how to use RabbitMQ with Kubernetes? Or how to set up a RabbitMQ cluster: https://www.rabbitmq.com/clustering.html? Or how rolling updates interact with RabbitMQ? Or something else?
I think you should be able to create one service and one replication controller per server, and then use the service DNS names in the cluster configuration file. This is the current approach used to run Zookeeper, also. We have a long-standing TODO to make this less verbose (https://github.com/GoogleCloudPlatform/kubernetes/issues/260), but the current approach should be straightforward. You do lose the ability to use a single kubectl rolling-update command to update the cluster, but it's also straightforward to update the instances individually.
The setup at the current employer has one set of back office functions on a Java platform and another group of functions on two separate .NET-based platforms. There is no overall architect.
The Java guys decided to go for Apache QPID and AMQP for messaging, presumably amongst themselves, with the .NET systems and other external systems.
.NET architecture involves WCF services hosted in IIS/WAS and Windows Server AppFabric.
Does anyone have any experience of AmqpBinding and IIS/WAS, if there are any possible pitfalls?
I think your first problem will be IIS/WAS/AppFabric because non HTTP services hosted in WAS have additional requirements for infrastructure which consists of additional process (listener) running usually as as a windows service and communicating with worker process. This process is responsible for receiving and sending messages and allows service activation in WAS. I don't think that the QPID project has the listener process already created. You will most probably have to implement the listener yourselves - check this sample for custom UDP activator.
I'm trying to figure out what is the best solution to work with rabbitmq cluster via wcf.
Current setup:
2 IIS web servers (act as message produces and post messages to queue via amqp wcf client).
2 servers with rabbitmq broker (clustered with mirrored queue, rabbit1 and rabbit2)
Windows service ( worker) with hosted amqp wcf service that listens to incoming messages.
Web role posts messages to rabbit1 node and worker listens to rabbit1 node too. If rabbit1 node fails system(both web and worker) should switch to rabbit2. And that's the question, how to implement this in more elegant way rather than handling connection failures in application code.
First and the only approach I see now is to use wcf4 routing backup endpoints feature. This way solves problem on client side(web role) only but doesn't solve problem on wcf service side(worker role).
One way is to create a wrapper around your service host, used for storing a list of connection strings (which can come from config).
Add a handler to the service faulted event, where you can close and reopen the host with a different connection string.