We just started using Mule a month back and so far it's been a good learning. Right now, we have quite a few flows implemented that integrates our disparate systems. One of the requirements for us is to execute at the end of each flow some clean-up code. More like a finalizer construct.
I am looking for a generic approach that I can follow for all our flows.
Note - If i add a step (where i can execute the clean-up code) to the end of a flow - there is no guarantee that that step will be executed after the completion of all previous steps (as some of the steps are run on different threads; and we don't want to run the entire flow on one synchronous thread). Is there any eventing mechanism in Mule that notifies subscribers at the completion of all steps in a flow? I am also unsure if mule flow life-cycle will be a right fit here. Please help.
Thanks.
Probably a good candidate for this are Mule Server Notifications:
http://www.mulesoft.org/documentation/display/current/Mule+Server+Notifications
Related
I implemented two Flows(Initiating and Responder Flow) that run every X minutes from my Service. They make use of both the send and receive method. When one Node goes down the Flow stays in memory and does not return or throw an Exception. This turns into a problem since after some time the node will crash because too many Flows are running at the same time.
My question is how can i timeout a Flow automatically after X TIME. Or how can i terminate a Flow from the CordApp at all?
Things that i have tried are:
kill flow from RPCOps - no use, since i want to delete it from the Cordapp
kill flow from shell - no use, since i want to delete it in the Code under certain conditions
Make Flow inherit from TimedFlow and set the parameters accordingly - The docs say that this never actually kills the Flow but just restarts it
The fundamental problem you are facing is that there are too many flows running at the same time. If you are using Corda Open Source v4.x, keep in mind that it is single threaded, so if you need to have a several flows running in parallel you should switch to Corda Enterprise, which is multi-threaded.
More specifically about the killing of flows. In your case it seems that the Responder flow has some issues, so causing the Initiating flow to wait forever for a response that never arrives. It is normal behavior. Since the flow is started from the Initiating flow, you need to kill that, you cannot kill the Responder flow from the Initiating one (which is what you are asking in this question).
What you want to do is also theoretically wrong. It would imply that, in a distributed network with Initiating and Responder flows running on two different nodes belonging to two different Organizations, the first one could control the operations of the other. This is not how DLTs/blockchains are supposed to work, it is instead normal in a standard client/server architecture.
The only way to kill the Initiating flow is from either the node terminal or from an external client using CordaRPC.
I'm struggling to understand how to implement Eventual Consistency with the exposed example of BacklogItems and Tasks from Vaughn Vernon. The statement I've understood so far is (considering the case where he splits BacklogItem and Task into separate aggregate roots):
A BacklogItem can contain one or more tasks. When all remaining hours from a the tasks of a BacklogItem are 0, the status of the BacklogItem should change to "DONE"
I'm aware about the rule that says that you should not update two aggregate roots in the same transaction, and that you should accomplish that with eventual consistency.
Once a Domain Service updates the amount of hours of a Task, a TaskRemainingHoursUpdated event should be published to a DomainEventPublisher which lives in the same thread as the executing code. And here it is where I'm at a loss with the following questions:
I suppose that there should be a subscriber (also living in the same thread I guess) that should react to TaskRemainingHoursUpdated events. At which point in your Desktop/Web application you perform this subscription to the Bus? At the very initialization of your app? In the application code? Is there any reasoning to place domain subscriptors in a specific place?
Should that subscriptor (in the same thread) call a BacklogItem repository and perform the update? (But that would be a violation of the rule of not updating two aggregates in the same transaction since this would happen synchronously, right?).
If you want to achieve eventual consistency to fulfil the previously mentioned rule, do I really need a Message Broker like RabbitMQ even though both BacklogItem and Task live inside the same Bounded Context?
If I use this message broker, should I have a background thread or something that just consumes events from a RabbitMQ queue and then dispatches the event to update the product?
I'd appreciate if someone can shed some clear light over this since it is quite complex to picture in its completeness.
So to start with, you need to recognize that, if the BacklogItem is the authority for whether or not it is "Done", then it needs to have all of the information to compute that for itself.
So somewhere within the BacklogItem is data that is tracking which Tasks it knows about, and the known state of those tasks. In other words, the BacklogItem has a stale copy of information about the task.
That's the "eventually consistent" bit; we're trying to arrange the system so that the cached copy of the data in the BacklogItem boundary includes the new changes to the task state.
That in turn means we need to send a command to the BacklogItem advising it of the changes to the task.
From the point of view of the backlog item, we don't really care where the command comes from. We could, for example, make it a manual process "After you complete the task, click this button here to inform the backlog item".
But for the sanity of our users, we're more likely to arrange an event handler to be running: when you see the output from the task, forward it to the corresponding backlog item.
At which point in your Desktop/Web application you perform this subscription to the Bus? At the very initialization of your app?
That seems pretty reasonable.
Should that subscriptor (in the same thread) call a BacklogItem repository and perform the update? (But that would be a violation of the rule of not updating two aggregates in the same transaction since this would happen synchronously, right?).
Same thread and same transaction are not necessarily coincident. It can all be coordinated in the same thread; but it probably makes more sense to let the consequences happen in the background. At their core, events and commands are just messages - write the message, put it into an inbox, and let the next thread worry about processing.
If you want to achieve eventual consistency to fulfil the previously mentioned rule, do I really need a Message Broker like RabbitMQ even though both BacklogItem and Task live inside the same Bounded Context?
No; the mechanics of the plumbing matter not at all.
I'm using dask.distributed.Client to connect to a remote Dask scheduler running that manages a bunch of workers. I am submitting my job using client.submit and keeping track of the returned Future:
client = Client("some-example-host:8786")
future = client.submit(job, job_args)
I want to be able to know if/when the job has been sent to and accepted by the scheduler. This is so that I can add some retry logic in cases when the scheduler goes down.
Is there an easy way to get confirmation that the scheduler has received and accepted the job?
Some additional points:
I realise that distributed.client.Future has a status property, but I'm hesitant to use it as it was not documented in the API.
I have tried using dask.callbacks.Callback but with no success. Any assistance with using callbacks with distributed.Client would be appreciated.
EDIT: I could also have the job post back a notification when it starts, but I would like to leave this approach as a last resort if the Client does not support this.
I know in BPMN there is just a "start event" for each pool. In my case I have a pool that can begin when a message is caught or because the actor decide to do it by his own decision.
How can I model that? I'm not sure I can use an event-based exclusive XOR.
Maybe a complex gateway?
As stated in many best practice how-tos, it is NOT RECOMMENDED to use multiple start events in a pool. BPMN specification 1.2 contains this note too:
9.3.2.
...
It is RECOMMENDED that
this feature be used sparingly and that
the modeler be aware that other readers of the Diagram may have difficulty
understanding the intent of the Diagram.
...
On the other side, the common rule for the case with omitted start event is
If the Start Event is not used, then all Flow Objects that do not have
an incoming Sequence Flow SHALL be instantiated when the Process is instantiated.
I assume this will be fair enough for the case of manual process start too. Even if the process has only message start event it will be correctly started because Message Start Event is a fair flow object with no incoming sequence flow and thus it complies to the above rule.
However, if you want to be 100% sure the process will go the way you want then the Event Based Exclusive Gateway (which is available since version 1.1) is your choice. Placing it before multiple different start events will make the process choose either of them for start.
Further explanation can be found in this blog.
Unlimited process instances
If you don't mind that during execution of your process the pool could be used multiple times (e. g. once started by a message and 3 times by an actor) then you can simply use multiple start events (BPMN 1.2 PDF Spec 9.3.2 page 37 allows this):
Single instance
If you can only allow a single run of the pool, you might have to instantiate it manually at the start of your execution and then decide whether to use it and when. Here is an example of how this can be done:
The Event-Based Gateway (Spec 9.5.2.4) will "decide" what to do with your pool:
If Actor decides to start or a message comes from the main pool, some actions will take place;
If the process is "sure" that additional pool will not be required, a signal is cast to terminate its instance.
I am looking for a pragmatic solution to do Integration testing of our Integration tier based on Mule.
This article here has some excellent pointers about it, but looks a tad outdated. I am reproducing an excellent idea from the article here
Keeping track of the delivery of messages to external systems. Interrogating all the systems that have been contacted with test messages after the suite has run to ensure they all received what was expected would be too tedious to realize. How to keep track of these test messages? One option could be to run Mule ESB with its logging level set to DEBUG and analyze the message paths by tracking them with their correlation IDs. This is very possible. I decided to follow a simpler and coarser approach, which would give me enough certitude about what happened to the different messages I have sent. For this, I decided to leverage component routing statistics to ensure that the expected number of messages where routed to the expected endpoints (including error messages to error processing components). Of course, if two messages get cross-sent to wrong destinations, the count will not notice that. But this error would be caught anyway because each destination will complain about the error, hence raising the count of error messages processed.
Using this technique when I test my integration tier I will not have to stand up all the external systems and can test the integration tier in isolation which would be great.
#David Dassot has provided a reference implementation as well, however I think it was based on Mule 2.X and hence I cannot find the classes in the Mule 3.X codebase.
Looking around I did find FlowConstructStatistics but this is flow specific statistics and I am looking for endpoint specific statistics.
I do agree that as a work around we could wrap all outbound endpoints within sub-flows and get this working, but I would like to avoid doing this ...
Any techniques that help query the endpoint for the number of calls made, payload passed through the endpoints would be great!
First take a look to JMX, perhaps what you need is available right there.
Otherwise, if looging is not enough, and upgrading to the enterprise version is not ok for you. Give it a try to the endpoint level notifications.