I am trying to learn messaging system. I have found that RabbitMq and NServiceBus are using together in few places. My questions are
If I am using the RabbitMQ then why do i need NServiceBus? and vice versa
What NServiceBus can do but RabbitMQ or Kafka cannot?
Can I use NServiceBus and kafka together? Or Apache-Kafka does not require NServiceBus
Years ago, I asked myself the same question. I was looking at NServiceBus to work with a different message queue, but the question was the same.
I decided not to use NServiceBus.
6 Month later, I realized I had re-built half of what NServiceBus did... only much more poorly.
The equivalent question of why would you need NServiceBus with RabbitMQ, is to ask why you would need the .NET Framework with ASP.NET MVC, or WinForms, or XAML, or any of the built-in libraries that .NET ships with, when you have the Common Language Runtime.
Shouldn't the CLR be enough, after all?
Of course not. Having the runtime on which code can execute - the MSIL interpreter and execution engine - is not nearly enough to be productive.
Sure, you can write command-line applications that take input and produce output. But try to build a real application without the common libraries - without the built-in SQL Server drivers; without any 3rd party controls or libraries. Build a Windows Desktop app without the System.Windows namespace.
You need those libraries to give you collections, and database access, and window objects and UI controls.
Similarly, RabbitMQ gives you everything you need to get started and working, but not enough to maintain productivity.
Sure, you can grab the .NET driver for RabbitMQ and start producing and consuming messages.
For a while, this will work just fine.
Pretty soon, you'll find yourself creating a wrapper around the driver, so you can reduce the amount of code you need to write.
Then you'll find yourself needing to deal with ack vs nack, and you'll create a simple API for that.
Then the need for dead-letter queues will pop up with nack calls, and you'll wrap that up in your API - simplified compared to the rabbitmq driver, of course.
Eventually, you'll want to deal with poison messages - messages that are malformed and causing exceptions. Once again, you don't want to write one-off code for this, so you'll write a library to handle it.
The list goes on and on.
6 months from now, you'll find yourself working with a half-written, barely specified, untestable library that only mimics the value and capabilities of NServiceBus (or MassTransit or whatever other service bus library you choose).
I won't say you have to use NServiceBus. And I would say you should learn how RabbitMQ works, without it. But once you get beyond the basics of sending and receiving messages, the value of NServiceBus and other service bus implementations, becomes very apparent, very quickly.
It seems there is community support for Kafka transport in NServiceBus now: https://docs.particular.net/nservicebus/kafka/
(haven't tried it myself yet).
Related
My team is currently in the initial stages of designing implementations using NServiceBus (v4, possibly v5) in a number of different contexts to facilitate integration between a number of our custom applications. However we would also like to utilize NServiceBus to raise business events triggered from some of our off-the-shelf third-party systems. These systems do not provide inherent messaging or eventing apis, so our current thinking is to hook into their underlying databases using triggers and potentially SQL Service Broker as a bridge to NServiceBus.
I've looked at ServiceBroker.net but that seems to use NServiceBus v2 or v3 api's, interfaces, etc., by creating a totally new ITransport. We're planning on using more recent versions of NServiceBus though, so this doesn't seem to be a solid option. Other somewhat similar questions here on SO (all from a few years ago) seem to be answered with guidance to simply use the SQL Transport. That uses table-based pseudo-queues instead of MSMQ, but what's not clear is if it is then advisable to have SQL triggers hand-craft NServiceBus message records and manually INSERT them into the pseudo-queue tables directly, or whether there would still be some usage of SQL Service Broker in the middle that somehow more natively pops the NServiceBus messages onto the bus. And if somehow using the SQLTransport is the answer, what would be best practice to bridge the messages over to the main MSMQTransport-based bus?
It seemed like there was some concerted movement on SQL Service Broker bridging over to NServiceBus several years ago, but was deprecated once the native NServiceBus SQLTransport was introduced. I feel like maybe I'm missing something in terms of the modern NServiceBus approach to generating data-driven events in a design that is more real-time than a looped polling design.
You may want to take a look at the Gateway feature. You should be able to run 2 different transports and use the Gateway feature to bridge the two via HTTP.
We have a similar system, although it's slightly easier in that we control the underlying databases and applications (i.e. not 3rd party) and the current proof of concept uses the ServiceBroker / SQLDependency / ServiceBus as part of its architecture.
If you go this route, I would also advise using triggers to populate a common table, then monitoring that.
I didn't know about ServiceBroker.Net until today, so can't comment. I also haven't looked at CLR stored procs / triggers; whether there's any possibilities there.
Somebody else asked a question about nServiceBus and ServiceBroker which I answered here which may be useful for anyone looking to implement this.
We are designing a distributed .NET application on W2008R2 servers and need to send a low volume of messages to multiple known nodes. We looked at RabbitMQ and liked it, but found that we hardly need any of its numerous features that would justify the additional complexity of a separate product (and let's not forget Erlang). It appears that good old MSMQ would actually satisfy our needs.
In looking around, there are signs that maybe MSMQ is no longer being used that much. Many postings about it are old and there don't appear to be any new books about it on Amazon, etc.
What we like about MSMQ is the fact that it's already part of the OS. However, when embarking on a major new system, is MSMQ really still a viable technology choice?
We have been using MSMQ with Win Server 2008 and now Win Server 2012. As you said it's part of a system and just works. If you are a Windows guy, use it without fear. It is not obsolete.
Just other frameworks/technologies build on top of mqmq. If you happy with the features, then why would you add an extra dependency on RabitMQ or ActiveMQ, apart from more stuff to learn, and more things to fail?
I'm currently trying to find the best message queue solution for an appharbor application. Most of the ones of looked at assume you have a windows environment with MSMQ and DTC installed, which I don't believe the appharbor environment provides.
I would like something that works well with ravendb, as that is the database we are using. Something who's only dependence is on raven would be ideal, especially if it integrates with our existing unit of work. Ie, when save changes is called in our controller action the messages are saved in the same transaction.
It would also need a host that works in a console application for background processing.
Ideally I would like something that "just works" in a development environment also. With raven, for example, we use the embedded mode while developing and I would like something that doesn't require installation.
I've looked at nServicebus, which seems to fail these conditions because it needs a transport (msmq, sql, etc) and much of the documentation is out of date.
I also looked at rhino service bus but there is a distinct lack of documentation and community. I'm also not sure if it can depend entirely on ravendb.
The others I looked at all seemed quite heavyweight and required installation and configuration to run in a development environment.
Edit: the other option, is to implement our own.
First of all, congratulations on being the 1000th NServiceBus question on StackOverflow!
Second, if you were to use SQL for persisting your business data, then you could run NServiceBus on top of that same SQL where all the messages go through tables (instead of queues) and then you wouldn't need the DTC.
Third, if you did want to go with RavenDB as your transport for NServiceBus, you would have to implement the ISendMessages and IReceiveMessages interfaces on top of it, but I believe that somebody in the community has already started working on that, so possibly you could join forces with them.
Finally, I wouldn't recommend writing your own ESB these days - not when there are so many good choices already out there. You mentioned the issues of community and documentation - those tend to be handled the worst when writing your own infrastructure.
I have a scientific program written in F# which I want to parallelize and run on 1 server with multiple processors (64) and for the future also in the cloud (Windows Azure?). The program will have a simple 1-1 communication between the nodes (no broadcast etc.).
If I used WCF, would it be as fast as MPI? What has MPI that WCF does not? There exists Pure MPI .NET written for WCF which puzzles me even more. I do not know if to use WCF or MPI.NET or Pure Mpi running on WCF.
PS: I guess that TPL is out of the game for 64 processors and more, right?
It is difficult to give a concrete answer, because it all depends on the specific aspects of your application, its current architecture (I suppose you already have some app) etc.
As you mention MPI and WCF, I assume that the application is written as several components that communicate with each other. The best way to structure this kind of application is to use F# agents.
As far as I understand, you want to run the application on a single server first. If you write it using agents, the agents can just communicate directly with each other (so you don't need MPI or WCF).
TPL should work well on a single-server (with lots of CPUs), but it will not scale to the distributed setting - you cannot run Task on another machine. However, you can use it inside individual components (e.g. agents) that will be distributed.
Regarding MPI vs. WCF - I don't have enough experience to answer that. However, if you use agent-based architecture, it should be easy to try various options. You may also check out fracture and related projects, which aims to implement high-performance sockets for F# (and possibly distributed agents in the future).
If you're doing it on 1 server you could just execute one process and execute the code in parallel. That way you could share memory more easily and faster than doing it through messages like MPI and WCF. Although the overhead of communication might not be that much, depending on your problem + solution.
Also the changes to your code would be much less that way, F# can usually be turned into prallel code with little effort. Going to MPI/WCF would require you to rewrite large portions.
Googling for F# + parallel gives plenty useful info that you should read first, like this for a good start:
http://blogs.msdn.com/b/dsyme/archive/2010/01/09/async-and-parallel-design-patterns-in-f-parallelizing-cpu-and-i-o-computations.aspx
So on 1 server, I woudl use the parallel features of F#, it's designed to prallelize easily.
Later when you want to go for cloud, that would be turning it into cleint-server. That's a different problem then parallization. I would treat and solve them seperately.
On the MPI vs WCF. WCF is designed as a RPC technology, i.e. you call remote procedures and get answers. If you want to use it for parallel programming with separate processes, you would have to create the boilerplate code for that. (Keep track of subsribed clients etc.)
MPI was designed to run that kind of architecture and handles it much more easily. (the first process gets number 0 and is the master, the other are slaves get numbered incrementally etc.)
Howver I don't think MPI will be very good to go cloud, since that invloves http, protocols, security etc. Not sure how well MPI works for those kind of things, WCF will handle that very well indeed.
The fact that there is an MPI.NET for WCF is because MPI is about a certain style of parallizing code that a lot of people are familiar with. So you can use the programming concepts and use them on the .NET platform leveraging WCF for the communications.
Something else you might want to look into if you need to exchange a lot of data over the wire is protocol-buffers (see protobuf-net for instance). That can easily be combined with WCF for communication and is very lean in serializing structured data so you can send over the wire efficiently.
Gert-Jan
WCF and MPI are different concepts. WCF is like a person A asks a person B to do something where as MPI is like a person A creates clones of himself (all clone have same ability/logic) and then these clones work on specific parts of the problem to be solved and once done they combine their results.
So choosing between which one fits your specific application depends on the problem your application is trying to solve. It may even be a combination of both WCF and MPI. Where your client application asks the WCF to do some task and the WCF create clones of the "problem solver" using MPI and when the clone are done with solving the problem (in parallel) they return the aggregated result back to the WCF and then that result is sent to client application.
You might also want to take at the 'mbrace' product, which provides a cloud monad (http://blogs.msdn.com/b/dsyme/archive/2011/08/23/m-brace-f-in-the-cloud.aspx). It's still at a fairly early stage though. I'm no expert but it may be that you can run an mbrace-based solution as effectively a private cloud on your 64-processor setup. When you outgrow that, a move to Azure would be seamless.
I fear I may be displaying my ignorance with this question, but here goes...
I would like to use WCF to implement interprocess communication between a .NET app and a third-party app written in Qt. The Qt app has a plugin architecture that, if I choose to, can be used to bootstrap some .NET classes to handle WCF cleanly at both ends, but I'd rather keep the codebase native and therefore I'm thinking of ways to make sure that whatever I send down the wire with WCF, I can reassemble at the other end using classes available in Qt.
Qt has a SOAP message class, so I figured the preferable solution - and the one that's closest to the one we've hacked together already - is to send SOAP messages and pick them up off a QLocalSocket. Question is, is it possible to force WCF to encode messages as SOAP over a NetNamedPipeBinding, and if so, is it wise to do so?
I'm feeling rather wary at this point that my question might not make complete sense due to my shaky understanding of the technology involved. If this is the case, please take the time to explain why instead of just saying 'no'.
edit #1: I figure an update is warranted, as I've investigated some and should report my findings.
Firstly, I have found that Qt is a pig. The QtSoapMessage class I mentioned, it turns out, doesn't exist in the current version, and is available only as an after-market source package that you have to compile yourself. It took me many hours of googling to find out why this wasn't working. The Qt documentation is utterly dreadful, Qt Creator is counterintuitive in the extreme, and I've all but lost patience with it so haven't pursued this idea any further as yet. Furthermore, it isn't obvious how exactly I am to pass the socket data into the soap message constructor, which takes a QDomDocument, whereas the API for reading XML from a socket uses a QXmlStreamReader or somesuch. There doesn't seem to be any conversion between them.
You actually have a different problem to the one you think you have.
WCF will by default exchange SOAP messages over the NetNamedPipeBinding.
However, the message exchange is layered over some Microsoft proprietary protocols for transaction flow, message framing and encoding, which means that if on the Qt side you pick up a byte stream directly from a QLocalSocket you will have a lot of work to do to implement these underlying protocols before you will be able to get at the SOAP infoset itself.
It is possible to configure the NetNamedPipe binding to remove some of these protocol layers, but not all of them - the framing protocol will always be there, for example.
You might like to read my blog for a lot more detail on this.