I'm working on proof-of-concept project designed to explore the benefits of offloading work from a NIO server to a message queue for backend processing. I'm using Grizzly for the NIO boilerplate stuff, and Spring Integration for the messaging (with JMS/ActiveMQ as the messaging implementation). Basically, what I want to do is this:
Client connection -> Server -> Server creates "work-to-be-done" message -> JMS/ActiveMQ
On the ActiveMQ message queue, a number of "workers" will be actively consuming these messages, processing them, and placing the result on another queue. The server is listening for "response messages" on that queue, and once a message is picked up it will execute the following:
Response queue -> Server serializes the message to something the client can understand -> back to the client
My immediate problem is my lack of understanding of Grizzly, specifically how to decouple the event handling from the messaging. The server has to create the work-to-be-done message in such a way that when the reply message comes back from the worker, the server knows who the client was (find the related FilterChainContext in Grizzly) in order to send the tcp message.
I might be able to use FilterChainContext.getAddress() and place that on the work message, but I'm not sure how to code a method which takes a peer address and a message and somehow sends that (FilterChainContext.write()) when it has no FilterChainContext.
I'm playing with the idea now of keeping a Map around, but I'm apprehensive about this approach because I don't want stuff to go stale in a map if something happens to the message during serialization or processing.
Ideas and suggestions are welcome.
-Michael
You could use the TCP adapters/gateways (which have an option to use NIO), together with custom (de)serializers. If you must use Grizzly, you could write a server connection factory implementation. In the case of the outbound adapter (or inbound gateway), the endpoint is registered as a 'TcpListener' (using the connectionId) and the SI message contains the IpHeaders.CONNECTION_ID header used to determine which connection gets the reply. When a connection closes, it is unregistered (removed from the map).
Related
Setting up a CMS consumer with a listener involves two separate calls: first, acquiring a consumer:
cms::MessageConsumer* cms::Session::createConsumer( const cms::Destination* );
and then, setting a listener on the consumer:
void cms::MessageConsumer::setMessageListener( cms::MessageListener* );
Could messages be lost if the implementation subscribes to the destination (and receives messages from the broker/router) before the listener is activated? Or are such messages queued internally and delivered to the listener upon activation?
Why isn't there an API call to create the consumer with a listener as a construction argument? (Is it because the JMS spec doesn't have it?)
(Addendum: this is probably a flaw in the API itself. A more logical order would be to instantiate a consumer from a session, and have a cms::Consumer::subscribe( cms::Destination*, cms::MessageListener* ) method in the API.)
I don't think the API is flawed necessarily. Obviously it could have been designed a different way, but I believe the solution to your alleged problem comes from the start method on the Connection object (inherited via Startable). The documentation for Connection states:
A CMS client typically creates a connection, one or more sessions, and a number of message producers and consumers. When a connection is created, it is in stopped mode. That means that no messages are being delivered.
It is typical to leave the connection in stopped mode until setup is complete (that is, until all message consumers have been created). At that point, the client calls the connection's start method, and messages begin arriving at the connection's consumers. This setup convention minimizes any client confusion that may result from asynchronous message delivery while the client is still in the process of setting itself up.
A connection can be started immediately, and the setup can be done afterwards. Clients that do this must be prepared to handle asynchronous message delivery while they are still in the process of setting up.
This is the same pattern that JMS follows.
In any case I don't think there's any risk of message loss regardless of when you invoke start(). If the consumer is using an auto-acknowledge mode then messages should only be automatically acknowledged once they are delivered synchronously via one of the receive methods or asynchronously through the listener's onMessage. To do otherwise would be a bug in my estimation. I've worked with JMS for the last 10 years on various implementations and I've never seen any kind of condition where messages were lost related to this.
If you want to add consumers after you've already invoked start() you could certainly call stop() first, but I don't see any problem with simply adding them on the fly.
Could someone with Pika experience give me a quick yes/no response as to whether the following functionality is possible, or whether my thinking that it is indicates a lack of conceptual understanding of Pika.
My desired functionality:
Python service (single threaded script) has one connection to my RabbitMQ broker using the SelectConnection adapter.
That connection has two channels.
Using one channel, A, the service declares a queue and binds to some exchange E1.
The other channel, B, is used to declares some other exchange, E2.
The service consumes messages from the queue via A.
It does some small processing of those messages, [possibly carries out CRUD operates through its connection to a MongoDB instance,] then publishes a message to exchange E2 via B.
I have read the Pika docs thoroughly, and have not found enough information to understand whether this is doable.
To put it simply - can a single python script both publish and consume via one selectconnection adapter connection?
Yes of course. You can achieve that in many ways (via the same connection, different connection, same channel, different channel etc.)
What I do when I have implemented this in the past is, I create my connection, get the channel and setup my consumer with it's delegate (function). When my consume message function is called I get the channel parameter that comes with it, which I sub-sequentially use to publish the next message to a different queue. If you don't want to use the same channel, you can simply setup another then.
I am using reliable delivery in mule flow. It is very simple case that takes message from JMS queue (ActiveMQ based), invokes several actions depending on it's content and, if everything is fine - delivers it into another JMS queue.
A flow is synchronized, both JMS queues are transactional (first BEGINS, second JOINS transaction), redelivery is used and DLQ for undelivered messages. Literally: I expect that all messages are properly either processed or delivered to DLQ.
For processing orchestration I am using Scatter/Gather flow control which works quite fine until I call external HTTP service using HTTP connector. When I use default threading profile it happens, that some messages are lost (like 3 of 5000 messages). They just disappear. No trace even in DLQ.
On the other hand, when I use custom profile (not utilizing thread) - all messages are getting processed without any problems.
What I have noticed is the fact, default threading profile utilizes 'ScatterGatherWorkManager', while custom uses 'ActiveMQ Session Task' threads.
So my question is: what is the possible cause of loosing these messages?
I am using Mule Server 3.6.1 CE Runtime.
by default scatter gather is setup for no failed routes you can define your own aggregation strategy to handle lost message
custom-aggregation-strategy
https://docs.mulesoft.com/mule-user-guide/v/3.6/scatter-gather
Using techniques as hinted at in:
http://msdn.microsoft.com/en-us/library/system.servicemodel.servicecontractattribute.callbackcontract.aspx
I am implementing a ServerPush setup for my API to get realtime notifications from a server of events (no polling). Basically, the Server has a RegisterMe() and UnregisterMe() method and the client has a callback method called Announcement(string message) that, through the CallbackContract mechanisms in WCF, the server can call. This seems to work well.
Unfortunately, in this setup, if the Server were to crash or is otherwise unavailable, the Client won't know since it is only listening for messages. Silence on the line could mean no Announcements or it could mean that the server is not available.
Since my goal is to reduce polling rather than immediacy, I don't mind adding a void Ping() method on the Server alongside RegisterMe() and UnregisterMe() that merely exists to test connectivity of to the server. Periodically testing this method would, I believe, ensure that we're still connected (and also that no Announcements have been dropped by the transport, since this is TCP)
But is the Ping() method necessary or is this connectivity test otherwise available as part of WCF by default - like serverProxy.IsStillConnected() or something. As I understand it, the channel's State would only return Faulted or Closed AFTER a failed Ping(), but not instead of it.
2) From a broader perspective, is this callback approach solid? This is not for http or ajax - the number of connected clients will be few (tens of clients, max). Are there serious problems with this approach? As this seems to be a mild risk, how can I limit a slow/malicious client from blocking the server by not processing it's callback queue fast enough? Is there a kind of timeout specific to the callback that I can set without affecting other operations?
Your approach sounds reasonable, here are some links that may or may not help (they are not quite exactly related):
Detecting Client Death in WCF Duplex Contracts
http://tomasz.janczuk.org/2009/08/performance-of-http-polling-duplex.html
Having some health check built into your application protocol makes sense.
If you are worried about malicious clients, then add authorization.
The second link I shared above has a sample pub/sub server, you might be able to use this code. A couple things to watch out for -- consider pushing notifications via async calls or on a separate thread. And set the sendTimeout on the tcp binding.
HTH
I wrote a WCF application and encountered a similar problem. My server checked clients had not 'plug pulled' by periodically sending a ping to them. The actual send method (it was asynchronous being a server) had a timeout of 30 seconds. The client simply checked it received the data every 30 seconds, while the server would catch an exception if the timeout was reached.
Authorisation was required to connect to the server (by using the built-in feature of WCF that force the connecting person to call a particular method first) so from a malicious client perspective you could easily add code to check and ban their account if they do something suspicious, while disconnecting users who do not authenticate.
As the server I wrote was asynchronous, there wasn't any way to really block it. I guess that addresses your last point, as the asynchronous send method fires off the ping (and any other sending of data) and returns immediately. In the SendEnd method it would catch the timeout exception (sometimes multiple for the client) and disconnect them, without any blocking or freezing of the server.
Hope that helps.
You could use a publisher / subscriber service similar to the one suggested by Juval:
http://msdn.microsoft.com/en-us/magazine/cc163537.aspx
This would allow you to persist the subscribers if losing the server is a typical scenario. The publish method in this example also calls each subscribers on a separate thread, so a few dead subscribers will not block others...
The client sends a lot of messages to the server from a single thread, over a single WCF channel.
The client sends the message with BeginMyMethod(x, b) as it does not wish to block while they get processed.
We have reliable messaging turned on, as we don’t wish to lose any messages, or have them get out of order.
However the messages are being despatched on multiple threads on the server, so are being process out of order.
We can’t have the server being single threaded, as we don’t wish a long running request from one client to block other clients.
So I just wish to process all the messages that come from a single client (over a single channel) in order with only one message from each cleint being processed at a time.
This would be easy for raw socket programming, however how to I get WCF to work as I wish?
I am now thinking that ConcurrencyMode.Reentrant does not behave well when used with InstanceContextMode.Single If I set use ConcurrencyMode.Single the messages are kept in order, but my call-backs deadlock.
(The test that gets the messages out of order has no callbacks and does not make any outgoing WCF calls, so I would expect ConcurrencyMode.Reentrant to behave the same as ConcurrencyMode.Single in that given test, but it does not)
I a not using any WCF config files, the code is:
serviceHost = new ServiceHost(this);
serviceHost.AddServiceEndpoint(
typeof(IAllEngineManagersAsyncCallbacks),
new NetTcpBinding(SecurityMode.None, true),
endPointAddress);
I have now worked round this problem by:
Changing all my call-backs from the server to the client to be OneWay
Using a dispatcher in the client before passing on any callback from the server, so client code never calls the sever from within a call-back
The client call-back object is marked with CallbackBehavior(UseSynchronizationContext=false, ConcurrencyMode=ConcurrencyMode.Single)
When running in Winform or WPF I use SynchronizationContext.Post to depatch the callbacks
When the cleint is a Console or a Windows server I use a custom depatcher.
So letting me use ConcurrencyMode.Single on both the server and the client.
It is now working as expected.
(BeginMyMethod(x, b) is still being used to send messaged from the client to the server)
(ConcurrencyMode.Reentrant seems to sometimes release the lock even when the WCF call is not made on the some thread that is processing the incoming message, it is just not a useful as Reentrant was in DCOM)
If you are using the generated BeginXXX async-methods, these are executed on a ThreadPool thread.
So although you've send the messages in a defined order, nobody guarantees you in which order the ThreadPool executes the requests.
I think that Reentrant mode implies that you allow messages be processed out of order. Normal behavior of such service would be: get message, put in queue for internal threads to process, and when it's done notify client about result. So maybe your service get messages in proper order but some of them are quiker to process and return earlier than others?