For a new auction system I am looking for which technology is the best for me.
When there is a new bid, I want to notify the listening users on the auction page. This is something for a pubsub technique, i presume.
First I did take a look at RabbitMQ, and I think this is a good way to build this. But it means I have an extra single point of failure.
So now I am leaning towards Redis PubSub. I know it has disadvantages, because when an user is not listening it won't re-send messages. But that is not a problem. When a user sign in it has all the current bids, and then only want updates. I don't plan to create a chat with a history.
What can you advice? Are there anymore disadvantages to use Redis for this? How about the stability? When a bid occurs, and I want to send the newest price to all listening users, how certain am I everyone gets the message?
Does anyone have experience with this situation?
Thanks
Pro: redis is much simpler than RabbitMQ to set up.
Cons: there is no guarantee of delivery with Redis.
I assume, that by "page" you mean standard HTML page with PHP on backend. If yes, then your main problem is not "should I use Redis or RabbitMQ", because you cannot make direct connection between your user browser and Redis or RabbitMQ.
First you have two answer to yourself, how will you provide updates for the page:
by regular ajax requests asking "is there any new for me"
by using some implementation of websocket
and after selecting answer, you will see that pub/sub mechanism has any use at all in your situation.
Related
I am looking to replace an in-house key-value store and dispatch system and I keep hearing that RabbitMQ may be a solution.
I understand that sends and receives messages using queues, and that these events are triggered by producers creating messages, and consumers receiving them.
But what happens if a consumer is created after a message was sent? Can the consumer ask the queue what its last message was? If not, do I need to include some sort of database to store these messages? Or am I looking for some other technology?
A use case is that I want a GUI to get/set parameters that are used by other apps on a local network. On initialization, the GUI needs to know what the last values were.
In an attempt to answer my own question, it may be that RabbitMQ is not what I am looking for. I may want to instead use Kafka which stores its latest key:value pair in a table. Or I may want to use Redis. What do you think?
Thank you for your assistance.
I think I found a satisfactory answer to my question. I'm looking to create a request-reply model, which RabbitMQ is quite capable of handling. Upon opening the GUI, it sends a request to some other process for some variable, stored either in memory or in a database. That process responds with the requested data. Easy enough.
this question is about best practices, and to figure out if I am on the right track. Maybe my thoughts are wrong, maybe there are better alternatives.
I use RabbitMQ (well - evaluate) for communication between services.
Some producders deliver data, some consumers consume it. Great.
For easy understanding let's assume these are messages containing customer addresses.
Later in development I want to introduce a new consumer, working on customer addresses. And here is my question:
Is it possible, to KEEP messages somewhere, so if there is a new consumer, it will get all data from the beginning? So the new consumer could work on that data, and for the user it seems as this new service was always in place.
Is this possible with rabbitmq? Where can I start? If not: what else would help such a solution?
I have used RabbitMQ, but am not an expert, so can't answer that aspect of your question.
However, what you are considering is very likely a bad idea. If you are using RabbitMQ as a queue, you should be thinking of it a means of communicating between services, not as a source of truth for information.
Somewhere in your enterprise, you (hopefully) have a service and accompanying database that stores customer address data. When a new service is deployed, it should consume the data from that service. This may be via migration script, API query or other means. Once the new service is up to date, it can then start processing new data via the queuing infrastructure.
When my web app subscribes to a Redis channel (mostly on Application_Start), it should automatically load the current channel content, but not wait for the next publish within this channel.
I couldn't find any way to achieve this - but as this "problem" appears to be so common and trivial I guess there must be an easy solution for this?
In the web app I'm using StackExchange.Redis (in case that's relevant). Who can help? Thx in advance!
The answer is no, there is no option to do this using Redis pub/sub functionality, Redis don't actually store messages which being published to the channel, so you can't retrieve them when you connect to channel.
Take a look at RabbitMQ with their persistent queues and message acknowledgements, which they have out of the box.
As there's obviously no comfortable option available in Redis, I'm now publishing the channel message also as a regular key-value. So the clients take it from key-value store before subscribing to the channel.
We have multiple web and windows applications which were deployed to different servers that we are planning to integrate using NservierBus to let all apps can pub/sub message between them, I think we using pub/sub pattern and using MSMQ transport will be good for it. but one thing I am not clear if it is a way to avoid hard code to set sub endpoint to MSMQ QueueName#ServerName which has server name in it directly if pub is on another server. on 6-pre I saw idea to set endpoint name then using routing to delegate to transport-level address, is that a solution to do that? or only gateway is the solution? is a broker a good idea? what is the best practice for this scenario?
When using pub/sub, the subscriber currently needs to know the location of the queue of the publisher. The subscriber then sends a subscription-message to that queue, every single time it starts up. It cannot know if it subscribed already and if it subscribed for all the messages, since you might have added/configured some new ones.
The publisher reads these subscriptions messages and stores the subscription in storage. NServiceBus does this for you, so there's no need to write code for this. The only thing you need is configuration in the subscriber as to where the (queue of the) publisher is.
I wrote a tutorial myself which you can find here : http://dennis.bloggingabout.net/2015/10/28/nservicebus-publish-subscribe-tutorial/
That being said, you should take special care related to issues regarding websites that publish messages. More information on that can be found here : http://docs.particular.net/nservicebus/hosting/publishing-from-web-applications
In a scale out situation with MSMQ, you can also use the distributor : http://docs.particular.net/nservicebus/scalability-and-ha/distributor/
As a final note: It depends on the situation, but I would not worry too much about knowing locations of endpoints (or their queues). I would most likely not use pub/sub just for this 'technical issue'. But again, it completely depends on the situation. I can understand that rich-clients which spawn randomly might want this. But there are other solutions as well, with a more centralized storage and an API that is accessed by all the rich clients.
I am a newbie to real-time application development and am trying to wrap my head around the myriad options out there. I have read as many blog posts, notes and essays out there that people have been kind enough to share. Yet, a simple problem seems unanswered in my tiny brain. I thought a number of other people might have the same issues, so I might as well sign up and post here on SO. Here goes:
I am building a tiny real-time app which is asynchronous chat + another fun feature. I boiled my choices down to the following two options:
LAMP + RabbitMQ
Node.JS + Redis + Pub-Sub
I believe that I get the basics to start learning and building this out. However, my (seriously n00b) questions are:
How do I communicate with the end-user -> Client to/from Server in both of those? Would that be simple Javascript long/infinite polling?
Of the two, which might more efficient to build out and manage from a single Slice (assuming 100 - 1,000 users)?
Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )
I hope this isn't a crazy question and won't get flamed right away. Would love some constructive feedback, love this community!
Thank you.
Architecturally, both of your choices are the same as storing data in an Oracle database server for another application to retrieve.
Both the RabbitMQ and the Redis solution require your apps to connect to an intermediary server that handles the data communications. Redis is most like Oracle, because it can be used simply as a persistent database with a network API. But RabbitMQ is a little different because the MQ Broker is not really responsible for persisting data. If you configure it right and use the right options when publishing a message, then RabbitMQ will actually persist the data for you but you can't get the data out except as part of the normal message queueing process. In other words, RabbitMQ is for communicating messages and only offers persistence as a way of recovering from network problems or system crashes.
I would suggest using RabbitMQ and whatever programming languages you are already familiar with. Since the M in LAMP is usually interpreted as MySQL, this means that you would either not use MySQL at all, or only use it for long term storage of data, not for the realtime communications.
The RabbitMQ site has a huge amount of documentation about building apps with AMQP. I suggest that after you install RabbitMQ, you read through the docs for rabbitmqctl and then create a vhost to experiment in. That way it is easy to clean up your experiments without resetting everything. I also suggest using only topic exchanges because you can emulate the behavior of direct and fanout exchanges by using wildcards in the routing_key.
Remember, you only publish messages to exchanges, and you only receive messages from queues. The exchange is responsible for pattern matching the message's routing_key to the queue's binding_key to determine which queues should receive a copy of the message. It is worthwhile learning the whole AMQP model even if you only plan to send messages to one queue with the same name as the routing_key.
If you are building your client in the browser, and you want to build a prototype, then you should consider just using XHR today, and then move to something like Kamaloka-js which is a pure Javascript implementation of AMQP (the AMQ Protocol) which is the standard protocol used to communicate to a RabbitMQ message broker. In other words, build it with what you know today, and then speed it up later which something (AMQP) that has a long term future in your toolbox.
Should I just build everything out with jQuery in the 'old school' paradigm and then identify which stack might make more sense? Just so that I can get the product fleshed out as a prototype and then 'optimize' it. Or is writing in one over the other more than mere optimization? ( I feel so, but I am not 100% on this personally )
This is usually called RAD (rapid application design/development) and it is what I would recommend right now. This lets you build the proof of concept that you can use to work off of later to get what you want to happen.
As for how to talk to the clients from the server, and vice versa, have you read at all on websockets?
Given the choice between LAMP or event based programming, for what you're suggesting, I would tell you to go with the event based programming, so nodejs. But that's just one man's opinion.
Well,
LAMP - Apache create new process for every request. RabbitMQ can be useful with many features.
Node.js - Uses single process to handle all request asynchronously with help of event looping. So, no extra overhead process creation like apache.
For asynchronous chat application,
socket.io + Node.js + redis pub-sup is best stack.
I have already implemented real-time notification using above stack.