I am trying to get a further understanding of message buses and one question that keeps coming up in my head is "how does the message get onto the bus?". Now, I assume there is a service (WCF, etc) of some sort that receives messages and puts them onto the bus. So then the other question I have is isn't this service then likely to be a bottleneck? I assume you would architect this service so that it can be easily scaled, such as through load balancing? Or would there be another way?
Also (sorry, it was originally only supposed to be one question), where would the routing tables be held that define where messages should go; in a database? Again, wouldn't this then be a potential bottleneck?
I am trying to look at this from a non product (BizTalk etc) or framework (NServiceBus, Mass Transit, etc) perspective. As if you were going to be writing this kind of thing from scratch. I want to get my head about what you are getting and the potential issues. I guess if you use BizTalk it has the message box for the routing tables, a notorious bottleneck in the past. I also see that you have the concept of "on ramps" with the ESB part of 2009. But as I said, I would like to think beyond a product and how people see it should be architected.
Many thanks for any insight.
One thing you might want to consider is that a Service Bus is something slightly different than just a Message Bus. In order to understand the difference, we need to look at what is a service in the SOA sense.
A WCF service isn't an SOA service - as it isn't necessarily autonomous (either at runtime, where it can be blocked by other WCF services it calls, or at design time, where it may require versioning when the WCF services it calls change).
Most of the technical questions you raise (scaling, routing, etc) are first and foremost addressed by the autonomy of the service in question. Only then does an ESB begin to make sense.
I understand that this doesn't provide much in the way of guidance, but you can try reading some of the stuff I've written on this topic (for the past 3 years) on my blog and in the articles I've published. Here's a good (and recent) one that can get you started in the right direction:
http://www.udidahan.com/2009/09/29/article-eda-soa-through-the-looking-glass/
Hope that helps in some way.
Related
this question is about best practices, and to figure out if I am on the right track. Maybe my thoughts are wrong, maybe there are better alternatives.
I use RabbitMQ (well - evaluate) for communication between services.
Some producders deliver data, some consumers consume it. Great.
For easy understanding let's assume these are messages containing customer addresses.
Later in development I want to introduce a new consumer, working on customer addresses. And here is my question:
Is it possible, to KEEP messages somewhere, so if there is a new consumer, it will get all data from the beginning? So the new consumer could work on that data, and for the user it seems as this new service was always in place.
Is this possible with rabbitmq? Where can I start? If not: what else would help such a solution?
I have used RabbitMQ, but am not an expert, so can't answer that aspect of your question.
However, what you are considering is very likely a bad idea. If you are using RabbitMQ as a queue, you should be thinking of it a means of communicating between services, not as a source of truth for information.
Somewhere in your enterprise, you (hopefully) have a service and accompanying database that stores customer address data. When a new service is deployed, it should consume the data from that service. This may be via migration script, API query or other means. Once the new service is up to date, it can then start processing new data via the queuing infrastructure.
I am just starting out learning NServicebus (and SOA in general) and have a few questions and points I need clarification on regarding how the solution is typically structured and common best practices:
The documentation doesn't really explain what an endpoint is. From what I gather it is a unit of
deployment and your service will have 1 or more endpoints. Is this correct?
Is it considered best practice to have one VS solution per Service you are developing? With a project for messages, then a project for each endpoint, and finally a project that is shared with the endpoints containing your domain layer?
From what I read, services are usually comprised of individual components. Can (or should) any component in the service access the same database, or should it be one database per component?
Thanks for any clarification or insight one can offer.
I will try and answer your questions the best i can...
I'm not sure about the term "Best Practices" i would consider the term "Best Thinking" or "Paradigm"
Q1: yes, an endpoint is a effectively a deployed process that consumes the messages of it's queue.
It's dose not have to belong to a single "Service" (logical) (In the case of a web endpoint for example), an endpoint can have one or more handlers deployed to it.
Q2: I would go with one solution (and later repo) per logical domain service, Inside a solution I would create a project per message handler, because as you scale you will need to move you handler between endpoints, or to their own endpoint depending on scale. Messages however are contracts, so i would put them in a solution/s maybe split commands and events. You may consider something like nuget to publish your message packages.
Q3: A "Service" is a logical composition of autonomous components, each is a vertical slice of functionality, so they can share the same database, but i would say that only one component has the authority to modify it's own data. I would always try to think what would happen when you need to scale.
Dose this make sense?
On my team at work, we use the IBM MQ technology a lot for cross-application communication. I've seen lately on Hacker News and other places about other MQ technologies like RabbitMQ. I have a basic understanding of what it is (a commonly checked area to put and get messages), but what I want to know what exactly is it good at? How will I know where I want to use it and when? Why not just stick with more rudimentary forms of interprocess messaging?
All the explanations so far are accurate and to the point - but might be missing something: one of the main benefits of message queueing: resilience.
Imagine this: you need to communicate with two or three other systems. A common approach these days will be web services which is fine if you need an answers right away.
However: web services can be down and not available - what do you do then? Putting your message into a message queue (which has a component on your machine/server, too) typically will work in this scenario - your message just doesn't get delivered and thus processed right now - but it will later on, when the other side of the service comes back online.
So in many cases, using message queues to connect disparate systems is a more reliable, more robust way of sending messages back and forth. It doesn't work well for everything (if you want to know the current stock price for MSFT, putting that request into a queue might not be the best of ideas) - but in lots of cases, like putting an order into your supplier's message queue, it works really well and can help ease some of the reliability issues with other technologies.
MQ stands for messaging queue.
It's an abstraction layer that allows multiple processes (likely on different machines) to communicate via various models (e.g., point-to-point, publish subscribe, etc.). Depending on the implementation, it can be configured for things like guaranteed reliability, error reporting, security, discovery, performance, etc.
You can do all this manually with sockets, but it's very difficult.
For example: Suppose you want to processes to communicate, but one of them can die in the middle and later get reconnected. How would you ensure that interim messages were not lost? MQ solutions can do that for you.
Message queueuing systems are supposed to give you several bonuses. Among most important ones are monitoring and transactional behavior.
Transactional design is important if you want to be immune to failures, such as power failure. Imagine that you want to notify a bank system of ATM money withdrawal, and it has to be done exactly once per request, no matter what servers failed temporarily in the middle. MQ systems would allow you to coordinate transactions across multiple database, MQ and other systems.
Needless to say, such systems are very slow compared to named pipes, TCP or other non-transactional tools. If high performance is required, you would not allow your messages to be written thru disk. Instead, it will complicate your design - to achieve exotic reliable AND fast communication, which pushes the designer into really non-trivial tricks.
MQ systems normally allow users to watch the queue contents, write plugins, clear queus, etc.
MQ simply stands for Message Queue.
You would use one when you need to reliably send a inter-process/cross-platform/cross-application message that isn't time dependent.
The Message Queue receives the message, places it in the proper queue, and waits for the application to retrieve the message when ready.
reference: web services can be down and not available - what do you do then?
As an extension to that; what if your local network and your local pc is down as well?? While you wait for the system to recover the dependent deployed systems elsewhere waiting for that data needs to see an alternative data stream.
Otherwise, that might not be good enough 'real time' response for today's and very soon in the future Internet of Things (IOT) requirements.
if you want true parallel, non volatile storage of various FIFO streams(at least at some point along the signal chain) use an FPGA and FRAM memory. FRAM runs at clock speed and FPGA devices can be reprogrammed on the fly adding and taking away however many independent parallel data streams are needed(within established constraints of course).
I have two applications, let us call them A and B. Currently A uses WCF to send messages to B. A doesn't need a response and B never sends messages back to A.
Unfortunately, there is a flaky network connection between the servers A and B are running on. This results in A getting timeout errors from time to time.
I would like to use WCF+MSMQ as a buffer between the two applications. That way if B goes down temporarily, or is otherwise inaccessible, the messages are not lost.
From an architectural standpoint, how should I configure this?
I think you might have inflated your question a bit with the inclusion of the word "architectural".
If you truly need an architectural overview of this issue from that high of a level, including SLA concerns, your SL will be as good as your MSMQ deployment, so if you are concerned about SL, just look at the documentation on the internet about MSMQ and SLA.
If you are looking more for the actual implementation from a code standpoint, this article is excellent:
http://code.msdn.microsoft.com/msmqpluswcf
It goes over a lot of the things you'll need to know, including how to setup MSMQ and how to implement chunking to get around MSMQ's 4MB limit (if this is necessary... I hope it's not).
Here's a good article about creating a durable and transactional queue that will cross machines using an MSMQ cluster: http://www.devx.com/enterprise/Article/39015/1954
Currently im developing a server for satelite monitoring of objects.
In its current state it is very efficient and stable in hi load scenarios.
Server must be able to handle 50+mln messages per day, or more if load balancing is used.
It consists of gps data gateway(singleton), "databroker"(singleton), which is responsible for persisting data, providing it on request, and alerting subscribers about new data, login service(per call) and client service(per session), responsible for subscription and working with web interface and rich client.
At this state i can forsee those problems:
Growing complexity if new services are added.
Tight coupling.
Hard to configure.
Lots of low level code for handling subscriptions etc in future bl services(reporting for example).
To solve those problems i want to use something like ServiceBus.
MS solutions are to expensive for our customers, NServiceBus is frightening me a bit, due to its open source origins(and Class1 in source code=D) and CIO asked to avoid using it.
So i decided to write my own simple bus, and encountered problems with adressing and subscribing diffirent types of services(singleton, per session, per call) and also there is a requirement for load balancing.
I found quite elegant solution for this: use "adapters" for bus - wcf services for incapsulating some specific issues of working with services - like loadbalancing. So bus will only see adapters and route messages between them and they will forward messages further. But im concerned about perfomance and whole idea..
Will be very grateful to hear thoughts about all of this stuff=)
PS Bus and adapters use MSMQ for communication between them, but other services can use http,tcp bindings.
PS2 Sorry for my english, its not my native language=)
I'm probably just bringing the dead back but if you'd still like to implement your own bus these links may come in handy (on design level):
http://msdn.microsoft.com/en-us/magazine/cc500646.aspx
http://msdn.microsoft.com/en-us/magazine/cc546553.aspx
I found answers for some my questions about how dynamically register subservices and route messages from client to them by central router service
You could try using "sql service broker" as a service bus
http://javiercrespoalvez.com/2009/03/using-sql-service-broker-in-net.html
The blog post also has links to other service buses you could try.
Thanks for all replyes=) I was able to convince bosses to use NServiceBus. (After creating a working bus prototype on weekend =). Now it fits quite well into the system=)