Could you, please, suggest - if I need maximum intersystem/language compatibility, while the main server will be implemented on .NET, could I use NServiceBus or MassTransit, or is it better to use pure RabbitMQ instead?
NServiceBus or MassTransit would give pretty nice abstracrion level, but the easiness of communication between different solutions and environments are critical. I'm thinking more of using pure RabbitMQ now, but would be really gratefull for pointing put some pros and cons if I'm taking it into account wrong.
If you have multiple applications with different languages internally that must send and receive messages I would not recommend NServiceBus or MassTransit. They require certain message headers that they themselves add. You would never be able to leverage all the functionality offered by these messaging frameworks. However, if internally you are all .NET and you have multiple applications that will use the messaging infrastructure then NServiceBus and MassTransit will add a lot of value.
Regarding interoperability with third parties. A strength and weakness of both NServiceBus and MassTransit is that you must send and receive messages as strongly typed classes or interfaces. These get serialised to JSON/XML/BSON etc and deserialised back to types again.
Because of this they require message headers that indicate the type of the message for deserialisation purposes. Without the message type headers they won't work.
Working with types is much easier but it can cause interoperability issues. When integrating with third parties that send XML or JSON messages with no types requires you to create a translation tier between your services and the third party.
You could translate to your own type that maps onto the XML/JSON or translate to a simple type that contains a string property for the XML/JSON. Either way, your translation tier would publish the messages internally using MassTransit/NServiceBus and so the messages would contain all the necessary headers to fully exploit all the functionality they offer.
For sending messages to the third party, the translation tier would translate the messages to the XML/JSON expected by the third party.
How much of your messaging system is involved with 3rd party/inter-system integrations? If the answer is very little then NServiceBus and MassTransit would be great options as they offer lots of great functionality.
If the answer is a lot, then they might still be good options. Having a translation tier will shield your internal services from being exposed to the requirements and changing schemas of your third parties. It will give you greater control and flexibility at a cost of extra moving parts.
Ultimately, the translation tier is not as complex as implementing the patterns offered out-of-the-box by NServiceBus and MassTransit. So I would seriously consider them as a viable option.
Some links regarding inter-operability
https://docs.particular.net/nservicebus/messaging/third-party-integration
http://masstransit-project.com/MassTransit/advanced/interoperability.html
Related
Coming from a .net developer's perspective, I've been recently introduced to BizTalk. I was expecting something like a series of Service References, auto-mapping classes and workflows. I really wasn't expecting heavy XSD use and I was surprised by the orchestration maps.
I just don't understand why it isn't more like a bunch of enterprise features built on a foundation of WCF.
Can anyone help me understand the idea behind how BizTalk was designed?
BizTalk can work with WCF services, but doesn't need to for some simple scenarios. It can also work in scenarios where custom non-WCF adapters are needed - it includes many useful ones out of the box, like for FTP, SFTP, File system access, POP3, Sharepoint, Azure ServiceBus, MSMQ, and MQSeries. Custom adapters can be written for legacy systems and services that don't expose WCF endpoints. There are many WCF adapters for cases where WCF is useful, and these adapters can be used and configured a bit more easily than drawing up a WCF Service from scratch. BizTalk can also expose its services as WCF endpoints.
The real power of BizTalk is in its server architecture, which allows for high availability, durable messaging, suspending and resuming messages, advanced debugging options, and rapid development of artifacts (like maps and orchestrations). It also provides for some powerful out of the box support for EDI, HL7, and WCF LoB integration work.
XML is at the heart and soul of the BizTalk messaging engine. This is good because XML is standardized and powerful; it's bad because XML is unwieldy at times, especially when dealing with larger messages and BLOBs.
ReceivePorts get the data into BizTalk's messaging engine (using adapters and receive locations). Send ports send the XML (or other) data out using the adapters mentioned above.
Maps use XSLT behind the scenes to transform the XML messages; it's possible to direct a map to use custom XSLT, or to use C#, VB, or JScript as well. However, for most trivial mapping tasks, the visual mapping interface allows for rapid development and testing of mappings between different message types. They can be called from receive ports, send ports, or orchestrations.
Orchestrations are more or less services that use the XLANGs language. When designed properly, they can provide very powerful processing of business logic and application handling, all with the above mentioned architectural features that BizTalk provides (durable messaging, high availability).
I look at it from a different perspective. BizTalk is more inline with Web/SOAP and cross-platform standards, Xml and now JSON, than WCF. BizTalk also supports a lot more protocols than WCF. BizTalk supports WCF, not the other way around.
That the WCF stack can build Contracts on and serialize/deserialize .Net classes is the custom approach. Keep in mind, WCF is just hiding all the Xml/Xsd from you, it's still there and is the same as what BizTalk uses.
BizTalk was designed and shipped before WCF as a reliable, cross-platform, multi-protocol integration engine. In terms of capabilities, the BizTalk stack as a whole is several orders of magnitude beyond WCF. In practice, we spend a lot of time in a BizTalk app working around the limitations of WCF.*
*For clarity, I'm referring to the OOB binding elements mostly and their application to actual implementations. WCF as a framework is perfectly serviceable.
My research indicates that BizTalk has remained largely unchanged since 2004, and thus would not experience the kind of technological convergence seen in other areas of the Microsoft stack. The reason for this appears to be because of a painful migration from BizTalk 2002 to 2004 that no one wants to replicate. Reminiscent, to me, of the many versions of the Entity Framework.
In 2010-2011, there was a "BizTalk is dead" movement, with promises that a combination of WCF, Workflow Foundation, and AppFabric on Azure would be the replacement tools. There has been little talk of it at all since 2012 -- it looks like the two technologies both had their unique pros and cons, but never would the two compete.
BizTalk has the strength of out-of-the-box throttling and disk persistence and an assortment of adapters that aren't standardized elsewhere (enterprise-iness). It's as if its stance is to tame an unwieldy beast. It appears to suffer, still, from taking advantage of scalability options that have come about in the last 10 years. The other stack is more along the lines of what I initially expected but lacking in enterprise-iness.
I don't quite have my head wrapped around BizTalk being described as a publish/subscribe model versus... some other model. Need to look more into that.
In conclusion, I don't like either technology set, and I think they're both in need of work.
Thanks to all who read this question and those who answered it. I know subjective answers aren't a big thing on stack overflow.
In the distributed message transaction world, am trying to understand the different parts that are involved in developing distributed systems. From what I understand you can design messaging system using enterprise bus backed with a message queue system. Why is it a good idea to use both? Can the same be achieved by programming against just the message queuing system? What are the advantages of using both together?
You certainly can code directly against the messaging infrastructure and you will find that there are pros and cons w.r.t. each transport. There are many decisions that you will need to make along the way, though, and this is where a service bus may assist.
Developing directly against the queuing system will inevitably lead to various abstractions that you will require to prevent duplication.
A service bus will provide opinions/implementations for:
Message delivery
exactly-once (distributed transactions - distributed transactions are not supported by all queuing systems)
at-least-once (non-transactional)
at-most-once (will probably require some transactional processing but you can get away with no distributed transactions)
Retrying failed messages
Request / Response
Message distribution
Publish/Subscribe (probably quite easy with RabbitMQ directly, not so much with MSMQ directly)
Message Idempotence
Dependency Injection
Some service bus implementations provide a framework for implementing process managers (called sagas by most). My current opinion is that a process manager needs to be a first-class citizen as any other entity is but that may change :)
Anyhow, if you as still evaluating options you could also take a look at my FOSS project: http://shuttle.github.io/shuttle-esb/
So a service bus may buy you quite a bit out-of-the-box whereas coding against the queues directly may be a bit of work to get going.
I can't comment directly on MassTransit, having only tinkered with it.
I use NServiceBus and am a fan of it. I think there are valid reasons for directly using queuing technology, but I think rolling your own ESB using MSMQ/RabbitMQ would cost a lot more than simply using a commercial product (or open source product e.g. MassTransit).
So do you need it? No. Will it make your life much easier if the features match your requirements? Absolutely.
We've been using StreamInsight for a while to process various events. In the search of a good ESB platform, we also wonder if we could use StreamInsight for the same purpose.
A couple of the key differences between StreamInsight and something like NServiceBus are
Max message/event size (StreamInsight is obviously limited in this area)
Pub/Sub (takes some work to make StreamInsight work)
Easily turn something into a WCF endpoint (NServiceBus is good for this)
But limitations aside, couldn't StreamInsight serve as a poor man's ESB? Or I'm just trying to fix a square peg in a round circle?
I have no experience with StreamInsight but took a look at the documentation briefly to see what it is all about. Based only on this look, it seems to have some powerful event based features, the ability to source events from certain systems using adapters and output adapters for the target system. It seems to be stream based with a query syntax perhaps with a more analytical capability than integration. Aspects of it remind me of EAI tools but my opinion is that process events is a nice feature of ESB it could be a bit of a square peg in a round hole. The various tools do overlap however.
There are many definitions of an ESB on Stackoverflow and elsewhere. I like the simple definition that Hohpe uses on the EAIPatterns website:
http://www.eaipatterns.com/MessageBus.html
The Bus products, including NServiceBus, do over some simiar features, but IMO they tend to offer a more dynamic and potentially fine grained way to integrate. In Nservicebus messages are usually C# POCO classes. Some can be large and it does support a databus. The bus messages are move from endpoints on the bus transactional through a transport. PubSub is built in and intentional. Rather than using adapters, NServiceBus uses a C# API that allows the code to integrate to the bus.
I hope this helps.
Regards,
Joe.
I'm just about getting into WCF; but from what I've read so far, like the sample scenarios I found on MSDN and some other sites, I can do all that with web services and applications that call those web services. So why the need for an elaborate layer like WCF?
Most of the comparisons I've googled for explain it more from a programming point of view. Still trying to find answers without much success as to when it makes business and of course programming sense to use the WCF layer as opposed to traditional application to web services model.
Anyone here with experience on both and can advice on how to go about choosing either web services or going the WCF way? What are those things that can't absolutely be done using just plain old web services called by applications and where the WCF layer will save the day.
You've fallen for the Microsoft trap of "it's just about web services" :-)
It's actually a lot more:
it's about service-oriented programming in general (not just web services - you can also write TCP/IP based services, MSMQ queue-based messaging and a lot more)
it's about unifying all the diverse programming models that existed so far (ASMX, Enterprise Services, DCOM, .NET remoting)
it's about providing a lot of ready-made and ready-to-use plumbing which can handle things like reliable messaging, transaction support, security in any shape or form you'd like, service discovery, and a lot more
it's about separating the service implementation from the details of how clients will call it and making this a configurable stack of protocols, encodings etc.
Sure - most of this stuff can be done in ASMX, or .NET remoting - but try to convert an ASMX web service to be callable in your intranet using TCP/IP and transport security... Many of those "older" technologies have a very intricate and direct link to how they're being used - you can't easily change that without changing the whole service code.
WCF separates all these "plumbing details" like what endpoint to call, what protocol to use to call it, how to handle security etc. out into a "WCF stack" that's configurable and composable, so you can easily switch your service XYZ to use HTTP allowing anonymous users to call it, to using TCP/IP with Windows credentials required - your service code won't change a bit - it's only configuration of the plumbing.
That to me is the most compelling reason for WCF - I can totally concentrate on my actual service code, and not pollute it with lots of plumbing stuff - how to handle transports and text encodings and all that. And I can easily change that and adapt to new requirements and needs in deployment without having to touch my actual service code.
Plus, the second major point is extensibility - most of the older technologies just had their one, set way of doing things and many didn't lend themselves to being extended. You had to either adapt to use it the way they did it - or forget about it. WCF has a vast and very intricate system for extending just about anything - you can create your own transport protocol (people have created UDP or SMTP based bindings), you can create your own message encoders (like I had to do to talk to a web service which could only understand ISO-8859-1 encoded messages), and you can extend just about anything else in WCF - all in an organized, well-documented, very stable and safe way.
So these two things - separating out plumbing into configurable layers, and extensibility to the maximum - are the most compelling reasons for me to use WCF.
Edit: Kobi's link above, is a far better answer than mine.
WCF is basically a better architecture for supporting communications. It breaks many dependencies such as hosting (not iis dependent), transport, security, addressing into plugin components, and allows customisation to a very high degree.
Yes you can do a lot with traditional technologies, however you can do more with WCF. If you don't need the features now then of course you can can continue with legacy technologies, however if you prefer you can opt for a better architecture now with an eye on the future but it comes at a cost of having to switch technologies now.
Take this example. If you have a legacy asmx web service, how easily can you offer the same service via an MSMQed endpoint? With WCF its as simple as adding new config settings.
I assume that you are not asking "why not just stick with SOAP/HTTP". WSF allows you to choose a number of different transports rather than just simple HTTP, but as you observe the WS-* technologies allow you to do all that. So I think you're asking why use a powerful but complex framework when the raw technolgies are not impossibly complex?
You could ask this same question of any Framework. You could just use the basic technologies and avoid the learning curve of adopting the framework.
Frameworks such as WCF do have a learning curve, but consider what happens if you don't use them:
You find that you write boiler-plate code for each service invocation. You then either accept duplication or begin to refactor and build your own libraries. Before long you've developed your own Framework, but it's not the same as anybody else's. So then any new team memeber has to learn your local framework, serious learning currve.
Note also that WCF addresses issues such as the monitoring of the deployed solution.
The biggest appeal to me is testability. Services are defined by a CLR interface, which is quite easy to mock inside a test harness. Some words of warning, however. With great flexibility comes some pain in the configuration process, along with a few "gotchas". An example of a gotcha is that WCF--adhering closely to a "best practice"--requires an active SSL connection in order to pass SOAP authentication credentials over HTTP. This hinders testing quite a bit.
I've inherited this really weird codebase where they've built an external web service over a bunch of internal web services just to add authentication/authorization using WS-Security, WS-Encryption, et al. Less than a month into this engagement, I'm already feeling the pain of coupling volatile components through rigid WSDL, esp considering some of them use WCF and other choose to go WSDL first. Managing various versions of generated proxies and wrappers at various levels is a nightmare!
I'll admit the design is over-complicated and could have been much better, but my question essentially is:
Would you ever build a web service just to provide a cross cutting concern over a bunch of services?
Would this be better implemented as web service handlers?
and lastly...
Would you categorize this under the Web Service Gateway pattern?
I saw that very thing being built one year ago. I almost cried when the team took months to build 4 web services, 2 of which simply wrapped other internal ones, using WCF and some serious encryption. The only reason they wrapped the internal ones was to change the potential error numbers coming back.
So, would I ever intentionaly do that? Nope.
Would it be better implemented as almost anything else? yep.
Would I categorize it under the WTF pattern? absolutely.
UPDATE:
One thing I just remembered is that there is an architecture called "Enterprise Service Bus" It's purpose is to provide a common interface into other SOA systems. This way it doesn't matter what the different applications use for their end point mechanisms (WCF, WSE 1/2/3, RESTful, etc).
BizTalk is one example of an ESB and there are many other off the shelf programs that can be used. Basically, your app passes a message to the ESB and it handles sending that message, in a reliable way, to the other systems as well as marshalling any responses back.
This also means that you could insulate other applications from many types of changes to the end points. Of course, if the new end points require additional information, then you'd have to modify the callers. However, if all they are changing is the mechanism then a good ESB would be able to handle those changes without impacting your app.
I have seen similar implementations if you are exposing the services to the outside world and if you need to tighten down the security..check this MSDN column..