My team is currently in the initial stages of designing implementations using NServiceBus (v4, possibly v5) in a number of different contexts to facilitate integration between a number of our custom applications. However we would also like to utilize NServiceBus to raise business events triggered from some of our off-the-shelf third-party systems. These systems do not provide inherent messaging or eventing apis, so our current thinking is to hook into their underlying databases using triggers and potentially SQL Service Broker as a bridge to NServiceBus.
I've looked at ServiceBroker.net but that seems to use NServiceBus v2 or v3 api's, interfaces, etc., by creating a totally new ITransport. We're planning on using more recent versions of NServiceBus though, so this doesn't seem to be a solid option. Other somewhat similar questions here on SO (all from a few years ago) seem to be answered with guidance to simply use the SQL Transport. That uses table-based pseudo-queues instead of MSMQ, but what's not clear is if it is then advisable to have SQL triggers hand-craft NServiceBus message records and manually INSERT them into the pseudo-queue tables directly, or whether there would still be some usage of SQL Service Broker in the middle that somehow more natively pops the NServiceBus messages onto the bus. And if somehow using the SQLTransport is the answer, what would be best practice to bridge the messages over to the main MSMQTransport-based bus?
It seemed like there was some concerted movement on SQL Service Broker bridging over to NServiceBus several years ago, but was deprecated once the native NServiceBus SQLTransport was introduced. I feel like maybe I'm missing something in terms of the modern NServiceBus approach to generating data-driven events in a design that is more real-time than a looped polling design.
You may want to take a look at the Gateway feature. You should be able to run 2 different transports and use the Gateway feature to bridge the two via HTTP.
We have a similar system, although it's slightly easier in that we control the underlying databases and applications (i.e. not 3rd party) and the current proof of concept uses the ServiceBroker / SQLDependency / ServiceBus as part of its architecture.
If you go this route, I would also advise using triggers to populate a common table, then monitoring that.
I didn't know about ServiceBroker.Net until today, so can't comment. I also haven't looked at CLR stored procs / triggers; whether there's any possibilities there.
Somebody else asked a question about nServiceBus and ServiceBroker which I answered here which may be useful for anyone looking to implement this.
Related
I am trying to learn messaging system. I have found that RabbitMq and NServiceBus are using together in few places. My questions are
If I am using the RabbitMQ then why do i need NServiceBus? and vice versa
What NServiceBus can do but RabbitMQ or Kafka cannot?
Can I use NServiceBus and kafka together? Or Apache-Kafka does not require NServiceBus
Years ago, I asked myself the same question. I was looking at NServiceBus to work with a different message queue, but the question was the same.
I decided not to use NServiceBus.
6 Month later, I realized I had re-built half of what NServiceBus did... only much more poorly.
The equivalent question of why would you need NServiceBus with RabbitMQ, is to ask why you would need the .NET Framework with ASP.NET MVC, or WinForms, or XAML, or any of the built-in libraries that .NET ships with, when you have the Common Language Runtime.
Shouldn't the CLR be enough, after all?
Of course not. Having the runtime on which code can execute - the MSIL interpreter and execution engine - is not nearly enough to be productive.
Sure, you can write command-line applications that take input and produce output. But try to build a real application without the common libraries - without the built-in SQL Server drivers; without any 3rd party controls or libraries. Build a Windows Desktop app without the System.Windows namespace.
You need those libraries to give you collections, and database access, and window objects and UI controls.
Similarly, RabbitMQ gives you everything you need to get started and working, but not enough to maintain productivity.
Sure, you can grab the .NET driver for RabbitMQ and start producing and consuming messages.
For a while, this will work just fine.
Pretty soon, you'll find yourself creating a wrapper around the driver, so you can reduce the amount of code you need to write.
Then you'll find yourself needing to deal with ack vs nack, and you'll create a simple API for that.
Then the need for dead-letter queues will pop up with nack calls, and you'll wrap that up in your API - simplified compared to the rabbitmq driver, of course.
Eventually, you'll want to deal with poison messages - messages that are malformed and causing exceptions. Once again, you don't want to write one-off code for this, so you'll write a library to handle it.
The list goes on and on.
6 months from now, you'll find yourself working with a half-written, barely specified, untestable library that only mimics the value and capabilities of NServiceBus (or MassTransit or whatever other service bus library you choose).
I won't say you have to use NServiceBus. And I would say you should learn how RabbitMQ works, without it. But once you get beyond the basics of sending and receiving messages, the value of NServiceBus and other service bus implementations, becomes very apparent, very quickly.
It seems there is community support for Kafka transport in NServiceBus now: https://docs.particular.net/nservicebus/kafka/
(haven't tried it myself yet).
In the distributed message transaction world, am trying to understand the different parts that are involved in developing distributed systems. From what I understand you can design messaging system using enterprise bus backed with a message queue system. Why is it a good idea to use both? Can the same be achieved by programming against just the message queuing system? What are the advantages of using both together?
You certainly can code directly against the messaging infrastructure and you will find that there are pros and cons w.r.t. each transport. There are many decisions that you will need to make along the way, though, and this is where a service bus may assist.
Developing directly against the queuing system will inevitably lead to various abstractions that you will require to prevent duplication.
A service bus will provide opinions/implementations for:
Message delivery
exactly-once (distributed transactions - distributed transactions are not supported by all queuing systems)
at-least-once (non-transactional)
at-most-once (will probably require some transactional processing but you can get away with no distributed transactions)
Retrying failed messages
Request / Response
Message distribution
Publish/Subscribe (probably quite easy with RabbitMQ directly, not so much with MSMQ directly)
Message Idempotence
Dependency Injection
Some service bus implementations provide a framework for implementing process managers (called sagas by most). My current opinion is that a process manager needs to be a first-class citizen as any other entity is but that may change :)
Anyhow, if you as still evaluating options you could also take a look at my FOSS project: http://shuttle.github.io/shuttle-esb/
So a service bus may buy you quite a bit out-of-the-box whereas coding against the queues directly may be a bit of work to get going.
I can't comment directly on MassTransit, having only tinkered with it.
I use NServiceBus and am a fan of it. I think there are valid reasons for directly using queuing technology, but I think rolling your own ESB using MSMQ/RabbitMQ would cost a lot more than simply using a commercial product (or open source product e.g. MassTransit).
So do you need it? No. Will it make your life much easier if the features match your requirements? Absolutely.
We've been using StreamInsight for a while to process various events. In the search of a good ESB platform, we also wonder if we could use StreamInsight for the same purpose.
A couple of the key differences between StreamInsight and something like NServiceBus are
Max message/event size (StreamInsight is obviously limited in this area)
Pub/Sub (takes some work to make StreamInsight work)
Easily turn something into a WCF endpoint (NServiceBus is good for this)
But limitations aside, couldn't StreamInsight serve as a poor man's ESB? Or I'm just trying to fix a square peg in a round circle?
I have no experience with StreamInsight but took a look at the documentation briefly to see what it is all about. Based only on this look, it seems to have some powerful event based features, the ability to source events from certain systems using adapters and output adapters for the target system. It seems to be stream based with a query syntax perhaps with a more analytical capability than integration. Aspects of it remind me of EAI tools but my opinion is that process events is a nice feature of ESB it could be a bit of a square peg in a round hole. The various tools do overlap however.
There are many definitions of an ESB on Stackoverflow and elsewhere. I like the simple definition that Hohpe uses on the EAIPatterns website:
http://www.eaipatterns.com/MessageBus.html
The Bus products, including NServiceBus, do over some simiar features, but IMO they tend to offer a more dynamic and potentially fine grained way to integrate. In Nservicebus messages are usually C# POCO classes. Some can be large and it does support a databus. The bus messages are move from endpoints on the bus transactional through a transport. PubSub is built in and intentional. Rather than using adapters, NServiceBus uses a C# API that allows the code to integrate to the bus.
I hope this helps.
Regards,
Joe.
I'm currently trying to find the best message queue solution for an appharbor application. Most of the ones of looked at assume you have a windows environment with MSMQ and DTC installed, which I don't believe the appharbor environment provides.
I would like something that works well with ravendb, as that is the database we are using. Something who's only dependence is on raven would be ideal, especially if it integrates with our existing unit of work. Ie, when save changes is called in our controller action the messages are saved in the same transaction.
It would also need a host that works in a console application for background processing.
Ideally I would like something that "just works" in a development environment also. With raven, for example, we use the embedded mode while developing and I would like something that doesn't require installation.
I've looked at nServicebus, which seems to fail these conditions because it needs a transport (msmq, sql, etc) and much of the documentation is out of date.
I also looked at rhino service bus but there is a distinct lack of documentation and community. I'm also not sure if it can depend entirely on ravendb.
The others I looked at all seemed quite heavyweight and required installation and configuration to run in a development environment.
Edit: the other option, is to implement our own.
First of all, congratulations on being the 1000th NServiceBus question on StackOverflow!
Second, if you were to use SQL for persisting your business data, then you could run NServiceBus on top of that same SQL where all the messages go through tables (instead of queues) and then you wouldn't need the DTC.
Third, if you did want to go with RavenDB as your transport for NServiceBus, you would have to implement the ISendMessages and IReceiveMessages interfaces on top of it, but I believe that somebody in the community has already started working on that, so possibly you could join forces with them.
Finally, I wouldn't recommend writing your own ESB these days - not when there are so many good choices already out there. You mentioned the issues of community and documentation - those tend to be handled the worst when writing your own infrastructure.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Could not find an answer on this question, so would like to initiate this:
Tibco EMS vs. MSMQ vs. MQ.
How do these 3 technologies compare?
Which one is better and in which kinds of scenarios?
Specifically, I think to use one of these in SOA environment (.NET + WCF), where the scenario will mature over time.
I have one additional specific interest in the performance, which is important to mention. So, if given a choice, performance is of a critical priority.
I would appreciate to have a comparison table for a clear picture.
Thanks!
EDIT:
I am concentrated on two parameters: performance and scalability.
Scalability - how do these technologies compare in terms of supported concurrent users' count? which can support more users? scenario does not matter, let's choose the scenario which is supported by all them - e.g. simple queues.
Performance - in exactly the same scenarios, which performs faster?
If you want to use WCF than non of them really matters. You will get most of them only when you use their direct API.
MSMQ - MS technology installed with every Windows installation. It is only transport technology with support for queues.
Tibco EMS - Tibco technology supporting both queues and topics (publish/subscribe). It is expensive and more suitable for enterprise scenarios. You will most probably need other Tibco tools and technologies as well to implement full SOA solution (Tibco ActiveMatrix product suite). .NET and WCF will be only apps connected to this infrastructure which is more designed for Java world. It runs on non Windows platforms as well and together with Tibco Business Works it offers connectors (adapters) to many LOB applications. I like APIs for Tibco products but I really don't like UIs of their tools.
IBM MQ - IBM technology supporting queues and it also somehow emulates topics (publish/subscribe). Again it is expensive commercial solution more suitable for enterprise scenarios where mainframes are involved - that is biggest MQ advantage - it runs "everywhere". But that is end of advantages. APIs for both Java and .NET are terrible. .NET API is full of bugs and it doesn't work as expected. IBM doesn't understand .NET libraries versioning which leads to terrible problems when moving your client application to machines with different MQ clients installed, etc.
Edit:
There were several question / comments about what problems MQ has? As few examples you can check my MQ questions. Not every question is actually an issue but you will find few of them pointing directly to bugs. Those issues can already be fixed in new MQ client versions but that doesn't mean there are no other. Generally I found MQ .NET API the most frustrating library I have ever used - it even beaten hated SharePoint.
On the other hand if you just need to send and receive some message and don't plan to do anything special or use low level features you should be OK. At the end the API is used for a while and common use cases should work - if you are not happy enough to hit regression bugs.
For a simple integration scenario - i.e. 2 applications interacting in a Point to point manner , no difference will be there. You would better check the support of each technology within your applications. And in that type of scenarios, you shouldn't be worried about performance as the messaging time shouldn't be the main issue. On the other hand, the real selection would be based on the target model for integrating your whole enterprise. For example,
- Are you doing any mediation functions - e.g: data transformation, protocol mapping ...etc
- Will you integrate systems in a point to point manner or you may consider having a Hub / ESB?
- Will you cover security aspects in your integration scenario (Authorization, authentication, auditing, encryption, certificate exchange ...etc)
Finally having such vision will give better understanding of what real constraints you've for your design. Personally, I would go for WCF only if I'm not expecting complex integration scenarios and I'm not willing to spend money on the solution. And I would go for IBM if I'm building a foundation for SOA. And will go to Tibco if I'm planning a Java based integration with a defined scope.
Again it is expensive commercial solution more suitable for enterprise scenarios
where mainframes are involved
Not sure why you mentioned mainframes. Many MQ enterprise customers don't have them.
IBM MQ - IBM technology supporting queues and it also somehow emulates
topics (publish/subscribe)
MQ v7.0.0 (released 2008) and onwards supports pub/sub topics as a native feature, there is no emulation involved.
APIs for both Java and .NET are terrible.
The MQ Classes for Java and JMS have evolved over 10+ years and are used heavily by thousands of enterprises.
.NET API is full of bugs and it doesn't work as expected.
The .Net API has been around for 7+ years over a few major releases of MQ. I would imagine that the obvious bugs would have been shaken out by now.
I am concentrated on two parameters: performance and scalability.
MQ has unlimited scalability. Performance is very good even with no tuning.
MQ is best only if you need to integrate with lots of mainframes. Pub/Sub is implemented poorly and the many APIs are 'strange to use'.
If all your applications are Windows, MSMQ might be a good choice, but it will be difficult to bridge into Unix or Java worlds.
The whole Java community standardized on JMS so TIBCO EMS is a good choice if you ever want to connect non-Windows applications.