I'm searching for a solution how to receive Amazon MWS notifications at a Rabbitmq instance. Can someone please help?
Currently, notifications can only be sent to an Amazon Simple Queue Service (Amazon SQS) queue.
source: http://docs.developer.amazonservices.com/en_US/subscriptions/Subscriptions_ReceivingNotifications.html
Related
Im quite new to AWS and also new to Kafka (using Confluent platform and .NET) .
We will receive large files (~1-40+Mb) to our S3-bucket and the consuming side of this should process these files. We will have all our messaging over Kafka.
Ive read that you should not send large files over Kafka, but maybe Im misinformed here?
If we instead want to just get an event that a new file has arrived on our S3-bucket (and of course some kind of reference to it), how would we go about?
You can receive notifications about events that happen in your S3 bucket like when a new object is created/deleted etc.
From the S3 documentation (as of writing this), the following destinations are supported:
Simple Notification Service (SNS)
Simple Queue Service (SQS)
AWS Lamdba function
For instance, you can choose SQS as your S3 notification destination and use Kafka SQS Source Connector to stream the events to Kafka.
Then you can write your Kafka consumer applications that react to this events.
And yes, it is not recommended to send large files over Kafka. Just send pointers to them and let the consumer application fetch the information using those pointers. If you are consumer wants to fetch some s3 objects, configure your consumer to use the S3 SDKs.
Useful resources:
Enabling event notifications in S3
S3 Notification Event Structure (JSON) with examples
Kafka SQS Source Connector
Assuming I have a Redis instance running on Google Memory store if there any way to send a Google PubSub notification on notify-keyspace-events events?
My ultimate goal is to trigger a cloud function subscribed to a topic that I can listen to on the event.
So far I've only been able to accomplish this using a dedicate node instance listing for the event.
Anyway to go around this? The documentation here doesn't mention anything related.
I have an exercise about Redis Pubsub like the following:
In case the publisher publish a message but the subscriber has not received the server has crashed. How to subscriber receive that message when restart server?.
Please help me, thank you !
In this case, the message is gone forever.
Redis only has limited support for PUBSUB scenario. Besides your case, if the connection between Redis and the client is lost, the client will also lose all published messages.
If you need more reliable PUBSUB tools, you should try other stuffs, e.g. Kafka, RabbitMQ.
The idea is:
I have N WCF services which connected and subscribed to the same Redis message channel. These services use this channel to exchange messages to sync some caches and other data.
How each service can ignore its own messages? I.e. how to publish to all but me?
It looks like Redis PUB/SUB doesn't support such filtration. So, the solution is to use set of individual channels for every publisher and common channel for subscription synchronization between them. Here is an golang example of no-echo chat application.
I am looking to implement rabbitmq on google compute engine to handle messages on my android and ios messaging app. I have heard that rabbitmq can be quite power hungry, so i am wondering what the best solution to combat this is?
Do i use a different protocol like MQTT or so i use something like GCM to handle the connection to and from the apps and let rabbitmq just handle queuing the messages?
You would never want make a direct connection from mobile device to your RabbitMQ server, especially if the app on the device is a consumer. RabbitMQ consumers have to poll RabbitMQ continuously to check if there are messages pending for them. You would want a web-server to handle actual HTTP POST/GET of messages from devices. The webserver will do two things:
Save the message to DB (along with the source and intended destination info)
queue APN/GCM push messages to a RabbitMQ (the broker here) exchange
you will need to build a daemon to monitor RabbitMQ for these push messages that have been queued. The daemon's sole task would be to connect or maintain a connection to Apple's or Google's push messaging services and notify your apps that they have a message pending. If a device is notified of a pending message, it contacts the webserver to consume the message