Is there a way to have a pub/sub queue in RabbitMq in which any of the subscribers could vote and all give a thumbs up (or more importantly a thumbs down) before processing continues?
I am not sure what to call this so It is very hard to research.
I am trying to make subscribers that can be added and have the ability to veto a process without knowing about them up front.
{edit below}
I am essentially trying to build a distributed set of services that could filter in very specific use cases as they are discovered. I am trying to do this so I do not have to down my service and version every time one of these new use cases is discovered.
Here is a contrived example but it gets the point across:
Lets say I want to calculate if a number is Prime.
I would like to have a service that has general easy rules, (is factor of 2? is factor of 3?)
But lets say that we are getting into very large numbers, and I find a new algorithm that is faster for finding specific cases.
I would like to be able to add this service, have it subscribe to the feed and be able to trigger "NotPrime" and have all other subscribers abandon their work as a result of the veto.
In a monolithic world I would look at some sort of plug in framework and have that implement the filters. That seems like mixing strategies in a bad way if were to do the same within a micro service.
Related
We are into the lead business. We capture leads and pass it on to the clients based on some rules. integration to each client very in nature like nature of the API and in some cases, data mapping is also required. We perform the following steps in order to route leads to the client.
Select the client
Check if any client-specific mapping(master data) is required.
Send Lead to nearest available dealer(optional step)
Call client api to send lead
Update push status of the lead to database
Note that some of the steps can be optional.
Which design pattern would be suitable to solve this problem. The motive is to simplify integration to each client.
You'll want to isolate (and preferably externalize) the aspects that differ between clients, like the data mapping and API, and generalize as much as possible. One possible force to consider is how easily new clients and their APIs can be accommodated in the future.
I assume you have a lot of clients, and a database or other persistent mechanism that holds this client list, so data-driven routing logic that maps leads to clients shouldn't be a problem. The application itself should be as "dumb" as possible.
Data mapping is often easily described with meta-data, and also easily data-driven. Mapping meta-data is client specific, so it could easily be kept in your database associated with each client in XML or some other format. If the transformations to leads necessary to conform to specific APIs are very complex, the logic could be isolated through the use of a strategy pattern, with the specific strategy selected according to the target client. If an extremely large number of clients and APIs need to be accommodated, I'd bend over backwards to make the API data-driven as well. If you have just a few client types (say less than 20), I'd employ some distributed asynchronicity, and just have my application publish the lead and client info to a topic corresponding to client-type, and have subscribed external processors specific for each client-type do their thing and publish the results on another single queue. A consumer listing to the results queue would update the database.
I will divide your problem statement into three parts mentioned below:
1) Integration of API with different clients.
2) Perfom some steps in order to route leads to the client.
3) Update push status of the lead to database.
Design patterns involved in above three parts:
1) Integration of API with different clients - Integration to each client vary in nature like the nature of the API. It seems you have incompitable type of interface so, you should design this section by using "Adapter Design Pattern".
2) Perform some steps in order to route leads to the client- You have different steps of execution. Next step is based on the previous steps. So, you should design this section by using "State Design Pattern".
3) Update push status of the lead to database: This statement shows that you want to notify your database whenever push status of the lead happens so that information will be updated into database. So, you should design this section by using "Observer Design Pattern".
Sounds like this falls in the workflow realm.
If you're on Amazon Web Services, there's SWF, otherwise, there's a lot of workflow solutions out there for your favorite programming language.
I have a huge network of data-collection servers which generate a large volume of real-time data.
In the past I've provided partners with the ability to get this data in near-real-time using HTTP GET's. But for many reasons I'm eager to ditch this.
So yeah... I'm eager to build out a new distribution system and I was thinking that a Message Queuing System was the way to go.
I need to be able to distribute data from my sources to a number of different partners. Some partners receive all of it, others just get a portion. And, if a partner gets disconnected, they need to be able to reconnect and not miss any data. (Although, for the sake of disk and memory I'd like their queued messages to expire after hour or so)
Lastly I need the system to be able to handle tens of thousands of enqueue's per minute.
Do you think Message Queuing is an appropriate scheme?
I was looking at using RabbitMQ. Is it difficult to maintain?
Thanks Very Much!
-Z
I cannot tell you if it is the right strategy in your specific case, but message products are indeed used in high message rate systems every day.
Much of the investment world uses various products, both commercial (Tibco) and Open source (ZeroMQ) to name just two, to handle market data from exchanges and other sources. These are likely at least as active as your data sensors are.
The publish/subscribe model, where some receivers want some messages and some receivers want all, along with late-join or other so-called guaranteed messaging are indeed standard features on most of these products.
So do go ahead and investigate products, I have not used RabbitMQ myself, so cannot comment on it specifically, however with a minimal abstraction layer, you should be able to insulate yourself from too many platform specific calls, and therefore allow you to swap message-bus implementers if the need arises. (You may even want to build such a shim as part of a proof-of-concept to test out more than one product for your specific purpose. You get experience in multiple products, flesh out the facade layer, and get up to speed on the products)
Good Luck
Is it possible to create end points dynamically at runtime. E.g. Send a message to a known endpoint with details of a new endpoint so that a network node can learn of new nodes on the fly.
NServiceBus does not support this out of the box, but if you really really want it (and you are sure that it is the right way to go), you are free to implement your own message routing and send messages explicitly to an endpoint with bus.Send(endpoint, message).
In a project I am currently involved with, we do this with great success, because it allows us to seamlessly sign services in and out of the system while it is running, resulting in zero downtime during upgrades.
It took a bit of work to get it working though, so I would only recommend this if you are certain that your requirements demand it.
On my team at work, we use the IBM MQ technology a lot for cross-application communication. I've seen lately on Hacker News and other places about other MQ technologies like RabbitMQ. I have a basic understanding of what it is (a commonly checked area to put and get messages), but what I want to know what exactly is it good at? How will I know where I want to use it and when? Why not just stick with more rudimentary forms of interprocess messaging?
All the explanations so far are accurate and to the point - but might be missing something: one of the main benefits of message queueing: resilience.
Imagine this: you need to communicate with two or three other systems. A common approach these days will be web services which is fine if you need an answers right away.
However: web services can be down and not available - what do you do then? Putting your message into a message queue (which has a component on your machine/server, too) typically will work in this scenario - your message just doesn't get delivered and thus processed right now - but it will later on, when the other side of the service comes back online.
So in many cases, using message queues to connect disparate systems is a more reliable, more robust way of sending messages back and forth. It doesn't work well for everything (if you want to know the current stock price for MSFT, putting that request into a queue might not be the best of ideas) - but in lots of cases, like putting an order into your supplier's message queue, it works really well and can help ease some of the reliability issues with other technologies.
MQ stands for messaging queue.
It's an abstraction layer that allows multiple processes (likely on different machines) to communicate via various models (e.g., point-to-point, publish subscribe, etc.). Depending on the implementation, it can be configured for things like guaranteed reliability, error reporting, security, discovery, performance, etc.
You can do all this manually with sockets, but it's very difficult.
For example: Suppose you want to processes to communicate, but one of them can die in the middle and later get reconnected. How would you ensure that interim messages were not lost? MQ solutions can do that for you.
Message queueuing systems are supposed to give you several bonuses. Among most important ones are monitoring and transactional behavior.
Transactional design is important if you want to be immune to failures, such as power failure. Imagine that you want to notify a bank system of ATM money withdrawal, and it has to be done exactly once per request, no matter what servers failed temporarily in the middle. MQ systems would allow you to coordinate transactions across multiple database, MQ and other systems.
Needless to say, such systems are very slow compared to named pipes, TCP or other non-transactional tools. If high performance is required, you would not allow your messages to be written thru disk. Instead, it will complicate your design - to achieve exotic reliable AND fast communication, which pushes the designer into really non-trivial tricks.
MQ systems normally allow users to watch the queue contents, write plugins, clear queus, etc.
MQ simply stands for Message Queue.
You would use one when you need to reliably send a inter-process/cross-platform/cross-application message that isn't time dependent.
The Message Queue receives the message, places it in the proper queue, and waits for the application to retrieve the message when ready.
reference: web services can be down and not available - what do you do then?
As an extension to that; what if your local network and your local pc is down as well?? While you wait for the system to recover the dependent deployed systems elsewhere waiting for that data needs to see an alternative data stream.
Otherwise, that might not be good enough 'real time' response for today's and very soon in the future Internet of Things (IOT) requirements.
if you want true parallel, non volatile storage of various FIFO streams(at least at some point along the signal chain) use an FPGA and FRAM memory. FRAM runs at clock speed and FPGA devices can be reprogrammed on the fly adding and taking away however many independent parallel data streams are needed(within established constraints of course).
I'm working on a 4-player network game in WPF and learning WCF in the process. So far, to handle the network communication, I've followed the advice from the YeahTrivia game on Coding4Fun game: I use a dualHttpBinding, and have use a CallbackContract interface to send back messages to clients. It works pretty well.
However, my service is getting very large. It has methods/callbacks to handle the game itself, but also the chat system, the login/registration process, the matchmaking, the roster/player info, etc. Both the server and client are becoming hard to maintain because everything is tied into a single interface. On the client for example, I have to redirect the callbacks to either the game page, the lobby page, etc, and I find that very tedious. I'd prefer being able to handle the game callbacks on the game page, the chat callbacks on the chat window, etc.
So what's the best way to handle that? I've thought of many things, but not sure which is best: splitting the service into multiple ones, having multiple "endpoints" on my service, or is there other tricks to implement a service partially where appropriate?
Thanks
You should have multiple components, each of which should be limited to one responsibility - not necessarily one method, but handling the state for one of the objects you're dealing with. When you have everything all in one service then your service is incredibly coupled to itself. Optimally, each component should be as independent as possible.
I'd say start with splitting it up where it makes sense and things should be MUCH more manageable.
I would support Terry's response - you should definitely split up your big interface into several smaller ones.
Also, you could possibly isolate certain operations like the registration and/or login process into simpler services - not knowing anything about your game, I think this could well be a simple non-duplex service that e.g. provides a valid "player token" as its output which can then be used by the other services to authenticate the players.
Multiple smaller, leaner interfaces also give you the option to potentially create separate, dedicated front-ends (e.g. in Silverlight or something) that would target / handle just certain parts of the whole system.
Marc