I am well experienced with the RabbitMQ and AMQP protocol, and have built a system with patterns for Commands, Requests and Events.
Now I am going to build a system running on AWS Lambda and therefore use SNS, SQS etc. I want to understand the "mapping" between these things.
What are the equivalent to an exchange in AMQP? What are the equivalent to a routing key?
How to set up queue bindings for fanout, direct and topic exchanges (or similar) in SNS and SQS?
How did other people handle this? To me it looks like RabbitMQ is a tool built to fit the usual needs of a message bus, where AWS provides blocks and you have to setup/build the functionality yourself. Am I right?
What are the equivalent to an exchange in AMQP?
The closest concept might be SNS, as you can configure a SNS topic to publish to n SQS queues. Then when you write to that topic, each subscribed queue gets a message. You can also write messages directly to SQS queues if you like.
What are the equivalent to a routing key?
There's no real equivalent for this. The SNS-to-SQS bindings don't allow for any additional filtering/control beyond topic-to-queue bindings. You could approximate routing by having multiple SNS topics i.e. each topic is a "routing key".
How to set up queue bindings for fanout, direct and topic exchanges (or similar) in SNS and SQS?
Fanout: write to SNS topic and every subscribed queue will receive the same message.
Direct: write directly to SQS queue(s), or SNS topic that has only those queues subscribed.
Topic: create SNS topic and subscribe queues accordingly.
How did other people handle this?
I used RabbitMQ before AWS messaging, so I went through the same learning curve. AWS doesn't provide as many exchange/routing bells & whistles, but in my experience you can get close enough with some combination of SNS topics and SQS queues.
It looks like the AWS IoT service with its MQTT provides what I need in order to do routing rules similar to the ones RabbitMQ provides!
Related
I would like to confront my understanding of google pubSub/lite vs RabbitMQ (using MQTT over WSS).
My use case is that I need something like a topic exchange. To send messages individually or to all or to some.
Having RabbitMQ I understand that I can create a topic have multiple queues linked via routingKey. E.g. amqTopic.routingKey1-10.
And I can push a message to a specific queue e.g. like this amqTopic.routingKey8
or push to the entire topic(all queues routed) like this amqTopic.*
Is it possible to create topic exchange structure with Google PubSub and if so how? I am not sure if I miss something. But from what I read I am inclined to say no, because google works like a direct exchange.
Thank you for helping..
This kind of topic exchange structure is possible to re-create using Cloud Pub/Sub filters. You can attach attributes to your messages when they are published (e.g. "routingKey": "8" or "routingKey": "all") and configure filters on your subscriptions to receive only messages meant for a particular routing key (attributes.routingKey="8" OR attributes.routingKey="all" in this scenario).
It's not currently possible to create this kind of topic exchange structure in Pub/Sub Lite.
I want to create a UI to see all the messages that are flowing through all exchanges in RabbitMQ server (of course other than the management console).
I am also using Mass Transit over rabbit but i am not sure if this matters.
Is this at all possible without having to code a consumer for each one of them one by one? If yes, any starting points?
The message exchanges used for publishing, as well as sending, are all bound to an exchange that has the same name as the queue for message delivery. So you could bind your own wire tap exchange on the broker to any queue exchange, and wiretap the messages to another queue of your choosing.
You can view the RabbitMQ topology layout in the documentation.
It was specifically done this way to make it easy to wiretap any endpoint, since all messages flow through a single fanout exchange.
This is a pretty broad question because it's not entirely obvious what you mean by "see", but regardless, you could create an observer on your bus. It's documented here and I think it's fairly straightforward: https://masstransit-project.com/MassTransit/usage/observers.html
In the observer you can handle various events when any message hits the MT message bus, and perform some kind of operation (like print the message, add logging, metrics, etc). If you have a microservice scenario it might be a good idea to add an observer to your shared library and add it to the bus in your individual applications.
I have a microservice architecture and now I need to introduce a notification center. Requirements are: any service is able to send a notification, any service is able to subscribe to any kind of notifications, UI (web) is able to subscribe to notifications (websockets are preferred). Of course I can write such service by myself but maybe there is ready-made robust solution for that.
UPD: I'm not looking for pub/sub messaging system as it is too low-level for notification center
What you are looking for is publish-subscriber messaging. If you are using AWS stack, then I can recommend Amazon SNS or Amazon SQS. I think Amazon SNS is more suitable because its push based.
Amazon SNS allows applications to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need
to periodically check or “poll” for updates.
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be
used to decouple sending and receiving components—without requiring
each component to be concurrently available.
Out of Amazon web services stack, there are a bunch of free messaging solutions:
RabbitMQ is one of the leading implementation of the AMQP protocol (along with Apache Qpid). Therefore, it implements a broker
architecture, meaning that messages are queued on a central node
before being sent to clients. This approach makes RabbitMQ very easy
to use and deploy, because advanced scenarios like routing, load
balancing or persistent message queuing are supported in just a few
lines of code. However, it also makes it less scalable and “slower”
because the central node adds latency and message envelopes are quite
big.
ZeroMq is a very lightweight messaging system specially designed for high throughput/low latency scenarios like the one you can find in
the financial world. Zmq supports many advanced messaging scenarios
but contrary to RabbitMQ, you’ll have to implement most of them
yourself by combining various pieces of the framework (e.g : sockets
and devices).
ActiveMQ is in the middle ground. Like Zmq, it can be deployed with both broker and P2P topologies. Like RabbitMQ, it’s easier to
implement advanced scenarios but usually at the cost of raw
performance.
Now you know what you need, I would recommend to read through each technology for a while and decide which one serves your goal more accurately. If that doesn't worth our time and your requirement is more specific & relatively small, then you can go for writing something on your own.
I come from a RabbitMQ background, and with RabbitMQ, you can set up exchanges that route messages to different queues based on a routing key.
In Kafka, how I am currently understanding topics is that they can be thought of as queues (that never get emptied). However, I am interested in putting different messages into different topics based on a certain criteria, and I would like to avoid doing that logic on the producer side.
Are there Kafka equivalent(s) to RabbitMQ's exchanges?
There is no equivalent. The only way to route different messages to
different topics is to put that logic on the producer side. Even deciding which partition of a topic to send an individual message to is left up to the producer.
Kafka's great strength is that it's really simple. That's part of why Kafka can scale really, really well. The downside is that Kafka doesn't have the feature set of a conventional message queue.
Kafka has something called a message key and when a message key added to the message for the first time Kafka pushes it to a random partition in the topic but when there is a new message with the same message key then Kafka pushes it to the same partition
I have multiple application instances inside of Amazon EC2, each running several worker processes. What I want is each worker process to be subscribed to some notification(e.g. configuration change). This notification should be basically broadcast message, so that once it is sent - every worker receives it.
I know SQS does not support messages broadcast. Looking through similar questions/threads I see the suggestions to use SNS instead of SQS. I'm not sure this will work for me due to the following reasons:
application instances are part of autoscaling group so they can be dynamically added and removed. In this case I don't see any clear way to unsubscribe every worker(I have multiple workers per instance) once instance gets terminated, which means I'll end up with the mess of dead subscribers after some time.
protocol to use for subscription is also not clear. HTTP endpoint looks like the only option, which means my every worker should run HTTP server on its own port. It also looks I should listen only on instance public IP, which adds one more layer of complexity and insecurity.
At the moment I have a solution based on third party - I'm using 0MQ pub/sub server. But I'm looking for some out-of-box solutions AWS provides.
Thanks,
Vovan
The out-of-the-box AWS solution that comes to mind would be to create one SNS topic, and then for each instance, when the instance boots up, it would create its own SQS queue and subscribe the queue to the SNS topic, so that each individual queue gets a broadcast copy of each message you publish to SNS.
You'd want unsubscribe and delete these queues on instance termination, which could be done with lifecycle hooks.
If you didn't want to use a server to manage the processing of the lifecycle hooks (which publish the launch or termination events to SNS or SQS) you could create an AWS API Gateway endpoint to fire an AWS Lambda function, then subscribe the API Gateway endpoint to the SNS topic using https, to handle the cleanup tasks in Lambda, with no server needed.
That's several services working together and may sound a little complicated, but would be very inexpensive and require little maintenance or attention.
One more solution I've figured out is to use Amazon Kinesis.The implication here is that each subscriber has to maintain it's own checkpoint to receive only most recent notifications.
I realize this is an old thread, but I'd like to share my experience with this. Kinesis has a 5 reads/sec throttle. So if you have 10 nodes polling for events in the stream 1/sec, you're going to be in a constant state of throttling.
Kinesis looks to be primarily for massive writes with just a few readers, which doesn't quite fit a broadcast to many nodes use-case.
Redis is handy solution for broadcasting a message to all subscribers on a topic. It is convenient because it can be used as a docker container for rapid prototyping, but is also offered by AWS as a managed service for multi-node clusters.