We have an application that uses BPM for managing long running processes.
We don't want to use this product anymore and we're looking at moving it to an ESB (i.e. Mule).
My view on this is that complex and long-running processes don't belong in an ESB. Also it would need to manage state which is not what an ESB should be doing in my opinion. An ESB is designed to handle high-volume, short-lived, real-time type messages? Am I correct in saying this?
Does anybody agree/disagree with this and what would the best solution be?
For example, should the BPM code be rewritten as a java application with a database behind it to manage state and use maybe quartz in Mule to handle periodic tasks to replace timers that are used in the BPM application?
I'd be interested to hear as many opinions on this as possible.
Many thanks.
I guess in your case you can move to an ESB if 3 characteristics are met:
the BPM solution was used for very low level messaging (EAI).
there is no much instrumentation of the data for business users, for example through a BAM module to represent KPIs and enforce SLAs.
there is no human interaction, for example through forms, workflows, approvals.
You can use ESB to move messages across systems. If you need to capture and track state, you probably want to use a database to keep it in sync. You can also enforce some level of transaction support via queue systems like JMS.
You probably need to have a good idea how to move things over. A good idea is to overlay the BPM with a thin layer, and then replace it without disrupting the user experience.
I hope this helps. Disclosure: I am the Chief Architect of Intalio, a BPM company.
I think I am a little late to the party here :), but I will write for the benefit of those who come here looking for an answer...
This is what Ross Mason has to say about the ESB BPM discussion.
Mule 3 provides strong orchestration capabilities with Flow, which is
great for short-lived transaction where the goal is to maximize
throughput and scalability. For other use cases like long running
transaction, Mule has support for commercial and open source BPM
products (like jBPM, Activiti, BonitaSoft BPM, etc.).
So yes ESB and BPM are complimentary solutions and not substitutes for each other.
To conclude, I guess you are correct about your observations.
Source
Related
My organization is looking to install MuleSoft to support data and process integration. We have 5 ERP's and need to consolidate data quickly for analysis and process improvements. Looking for references or issues you have experience with MuleSoft.
While seeking extra details from this curious person, I have got an interesting research paper putting up light on Enterprise Integration Architecture while leveraging the tools of Middleware like Oracle SOA Fusion Middleware and MuleSoft ESB specifically. There are many other tools that are fairly doing well in the market and yes are available as open source and at fairly high price in terms of yearly licencing cost. Coming back to main point. You get what you ask for!
First, Kindly go through the relevant MuleSoft Documentation to get started. Perform few POC's and observe how user friendly the tool is ! The tool is not too generic to quote and unquote standard inherent issue. It's the developer's ability which makes the tool to be used efficiently.
When you talk about Middleware Integration Services, you should have a background story of all the tools which falls under such category. Have you explored the other options ! such as spring boot micro services, dell boomi, web methods before.
Intro:
I'm working for a contractor company. We're making SW for different corporate clients, each with their own rules, SW standards etc.
Problem:
The result is, that we are using several bug-tracking systems. The amount of tickets flow is relatively big and the SLA are deadly sometimes. The main problem is, that we are keeping track of these tickets in our own BT (currently Mantis) but we're also communicating with clients in theirs BT. But as it is, two many channels of communication are making too much information noise.
Solution, progress:
Actual solution is an employee having responsibility for synchronizing the streams and keeping track of the SLA and many other things. It's consuming quite a large part of his time (cca 70%) that can be spend on something more valuable. The other thing is, that he is not fast enough and sometimes the sync is not really synced. Some parts of the comments are left only on one system, some are lost completely. (And don't start me at holidays or sickness, that's where the fun begins)
Question:
How to automate this process: aggregating tasks, watching SLA, notifying the right people etc. partially or all together?
Thank you, for your answers.
You need something like Zapier. It can map different applications and synchronize data between them. It works simply:
You create zap (for example between redmine and teamwork).
You configure mapping (how items/attributes in redmine maps to items/attributes in teamwork)
You generate access tokens in both systems and write them to zap.
Zapier makes regular synchronization between redmine and teamwork.
But mantis is not yet supported by Zapier. If all/most of your clients BT are in Zapier's apps list, you may move your own BT to another platform or make a request to Zapier for mantis support.
Another way is develop your own synchronization service that will connect to all client's BTs as each employee using login/password/token and download updates to your own BT. It is hard way and this solution requires continious development to support actual virsions of client's BTs.
You can have a look a Slack : https://slack.com/
It's a great tools for group conversations
Talk, share, and make decisions in open channels across your team, in
private groups for sensitive matters, or use direct messages
one-to-one.
you can have a lot of integrations tools, and you can use Zapier https://zapier.com/ with it to programm triggers.
With differents channels you can notifying the right people partially or all together in group conversation :)
The obvious answer is to create integrations between all of the various BT's. Without knowing what those are, it's hard to say if that's entirely possible. Most modern BTs have an API and support integrations. Some, especially more desktop based ones, don't. For those you probably have to monitor a database directly.
Zapier, as someone already suggested, is a great tool for creating integrations and may already have some of the ones yo need available. I love Slack and it has an API, but messages are basically just text and unless you want to do some kind of delimiting when you post messages to its API, it probably isn't going to work.
I'm not sure what budget is, but it will cost resources to create the integrations. I'd suggest that you hire someone to simply manage these. Someone who's sole responsibility is to cross-populate the internal and the external bug tracking system and track the progress in each. All you really need is someone with good attention to detail for this, they don't have to be a developer. This should be more cost effective than using developer resources on this.
The other alternative is simply to stop. If your requirements dictate that you use your clients' bug tracking software for projects you do for them, just use their software and stop duplicating the effort. If you need some kind of central repository or something for managing work maybe just a simple table somewhere or spreadsheet with the client, the project, the issue number, the status and if possible a link to the issue in the client's BT. I understand the need and desire for centralizing this, but if it's stifling productivity, then the opportunity costs are too high IMO.
If you create an integration tool foe this, you will indeed have a very viable product. This is actually a pretty common problem.
In the distributed message transaction world, am trying to understand the different parts that are involved in developing distributed systems. From what I understand you can design messaging system using enterprise bus backed with a message queue system. Why is it a good idea to use both? Can the same be achieved by programming against just the message queuing system? What are the advantages of using both together?
You certainly can code directly against the messaging infrastructure and you will find that there are pros and cons w.r.t. each transport. There are many decisions that you will need to make along the way, though, and this is where a service bus may assist.
Developing directly against the queuing system will inevitably lead to various abstractions that you will require to prevent duplication.
A service bus will provide opinions/implementations for:
Message delivery
exactly-once (distributed transactions - distributed transactions are not supported by all queuing systems)
at-least-once (non-transactional)
at-most-once (will probably require some transactional processing but you can get away with no distributed transactions)
Retrying failed messages
Request / Response
Message distribution
Publish/Subscribe (probably quite easy with RabbitMQ directly, not so much with MSMQ directly)
Message Idempotence
Dependency Injection
Some service bus implementations provide a framework for implementing process managers (called sagas by most). My current opinion is that a process manager needs to be a first-class citizen as any other entity is but that may change :)
Anyhow, if you as still evaluating options you could also take a look at my FOSS project: http://shuttle.github.io/shuttle-esb/
So a service bus may buy you quite a bit out-of-the-box whereas coding against the queues directly may be a bit of work to get going.
I can't comment directly on MassTransit, having only tinkered with it.
I use NServiceBus and am a fan of it. I think there are valid reasons for directly using queuing technology, but I think rolling your own ESB using MSMQ/RabbitMQ would cost a lot more than simply using a commercial product (or open source product e.g. MassTransit).
So do you need it? No. Will it make your life much easier if the features match your requirements? Absolutely.
We've been using StreamInsight for a while to process various events. In the search of a good ESB platform, we also wonder if we could use StreamInsight for the same purpose.
A couple of the key differences between StreamInsight and something like NServiceBus are
Max message/event size (StreamInsight is obviously limited in this area)
Pub/Sub (takes some work to make StreamInsight work)
Easily turn something into a WCF endpoint (NServiceBus is good for this)
But limitations aside, couldn't StreamInsight serve as a poor man's ESB? Or I'm just trying to fix a square peg in a round circle?
I have no experience with StreamInsight but took a look at the documentation briefly to see what it is all about. Based only on this look, it seems to have some powerful event based features, the ability to source events from certain systems using adapters and output adapters for the target system. It seems to be stream based with a query syntax perhaps with a more analytical capability than integration. Aspects of it remind me of EAI tools but my opinion is that process events is a nice feature of ESB it could be a bit of a square peg in a round hole. The various tools do overlap however.
There are many definitions of an ESB on Stackoverflow and elsewhere. I like the simple definition that Hohpe uses on the EAIPatterns website:
http://www.eaipatterns.com/MessageBus.html
The Bus products, including NServiceBus, do over some simiar features, but IMO they tend to offer a more dynamic and potentially fine grained way to integrate. In Nservicebus messages are usually C# POCO classes. Some can be large and it does support a databus. The bus messages are move from endpoints on the bus transactional through a transport. PubSub is built in and intentional. Rather than using adapters, NServiceBus uses a C# API that allows the code to integrate to the bus.
I hope this helps.
Regards,
Joe.
I am not a senior programmer but I have been deploying applications for a while and devloped small complete systems.
I am starting to hear about queueing systems such as RabbitMQ. May be, I never developed any systems that had to use a queueing system. But, I am worried if I am not using it because I have no idea what to do with this. I have read RabbitMQ tutorial on their site but I am not sure why I would use this for. I am not sure if any of those cannot be achieved by conventional programming with no additional component and regular databases or similar.
Can someone please explain why I would use a queueing system with a small example. I mean not a hello world example, but a a practical scenario.
Thanks a lot for your time
RM
One of the key uses of middleware like message queues is to be able to send data between non homogenous systems. The messages themselves can be many things. Strings are the easiest to be understood by different languages on different systems but are often less useful for transferring more meaningful data. As a result JSON and XML are very popular for the messages. These are just structured strings that can be converted into objects in the language of choice at the consumer end.
Additional useful features:
In some MQ systems like RabbitMQ (not true in all MQ systems) is that the client handles the communication side of things very nicely.
The messages can be asynchronous. If the consumer goes down, the messages will remain until the consumer is back online.
The MQ system can be setup to varying degrees of message durability. They can be removed from the queue once read or remain until the are acknowledged. They can be persistent so even if the MQ systems goes down message will not be lost.
Here goes with some possibly contrived examples. A Java program on a local system wants to send a message to a system on the connected through the internet. The local system has a server connected to the internet. Everything is blocked coming from the internet except a connection to the MQ. The Java program can publish the message to the MQ with out needing access to the internet. The message sits on the queue until the external system picks it up. The Java program publishes a message, lets say XML, and the consumer could be a Perl program. As long as they have some way of understanding the XML with a predefined way of serialization and deserialization it will be fine.
MQ systems tend to work best in "fire-and-forget" scenarios. If an event happens and others need to be notified of it, but the source system has no need for feedback from the other systems, then MQ might be a good fit.
If you understand the pros and cons of MQ and still don't understand why it would be a good fit for a particular system, then it probably isn't. I've seen systems where MQ was used but not needed, and the result was not pretty.
Most of the scenarios I've seen where it's worked out well is integration between unrelated systems (usually out-of-the-box type system). Let's say you have one system that takes orders, and a different system that fills the orders and ships them. In that scenario, the order system can use a MQ to notify the fulfillment system of the order, but the order system has no interest in waiting until the fulfillment system receives the order. So it puts a message in a queue keep going.
This is a very simplified answer, but it gives the general ideas.
Let's think about this in terms of telephone vs. email. Pretend for a minute that email does not exist. To get work done, you must phone everyone. When you communicate with someone via telephone, you need to have them at their desk in order to reach them (assume they are in a factory and can't hear their cell phone ring) :-) If the person you wish to reach isn't at the desk, you are stuck waiting until they return your call (or far more likely, you call them back later). It's the same with you - you don't have any work to do until someone calls you up. If multiple people call at once, you don't know about it because you can only handle one person at a time.
However, if we have email, it is possible for you to "queue" your requests with someone else, to answer (but more likely ignore) at their convenience. If they do ignore your email, you can always re-send it. You don't have to wait for them to be at the desk, and they don't have to wait until you are off the phone. The workload evens out and things run much more smoothly. As an added bonus, you can forward messages that you don't want to deal with to your peons.
In systems engineering, we use the term "closely coupled" to define programs (or parts of programs) that work like the telephone scenario above. They depend very closely upon each other, often sharing implementations among various parts of the program. In these programs, data is processed in serial order, one at a time. These systems are typically easy to build, but there are a few important drawbacks to consider: (1) changing any part of the program likely will cause cascading changes throughout the code, and this introduces bugs; (2) the system is not very scalable, and typically must be scrapped and rebuilt as needs grow; (3) all parts of the system must be functioning simultaneously or the whole system will not work.
Basically, closely-coupled programs are good if the program is very simple or if there is some specialized reason to use a closely-coupled program.
In the real world, things are much more complex. Programs cannot be that simple, and it becomes a nightmare to develop enterprise applications in a closely-coupled manner. Therefore, we use the term "loosely-coupled" to define large systems that are composed of many smaller pieces. The pieces have very well-defined boundaries and functions, so that changing of the system may be accomplished more easily. It is the essence of object-oriented design. Message queues (like RabbitMQ) allow email-like communication to take place among various programs and parts of programs, thus making workflow much more like it would be with people. Adding extra capacity then becomes a simple matter of starting up and additional computer wherever you need it.
Obviously, this is a gross simplification, but I think it conveys the general idea. Building applications that use message queuing enables you to deploy massively scalable applications leveraging cloud service providers. Here is an article that talks about designing for the cloud:
http://blogs.msdn.com/b/silverlining/archive/2011/08/23/designing-and-building-applications-for-the-cloud.aspx