I am writing some automation ruby scripts to test the streams and messages in queue.
My stream looks like
source| processor| processor| sink
Spring xd creates 3 internal queues.Now i have written ruby consumer to hook to these internal queues to test messages after every module(source/sink/processor).I see that when I post 100 message to my source Some message are getting lost in my ruby client consumer because they are flowing in stream .I think the message are going to the next module ie processor .Is there any way i can get all the message in my ruby client and test them and then I can make those message flow through the stream.Is there a way such that multiple consumers can receive same message.?
If you are registering other consumers on the bus queues, they will compete with the modules for messages.
I suppose that you are looking for a tap: http://docs.spring.io/spring-xd/docs/current/reference/html/#_taps
Hope this helps,
Marius
Related
My requirement is to clear all the messages from queue before processing the flow or publishing anything in the queue.
We are using rabbitMQ and due to some reason messages are stucked in the queue and because of that we are facing some issue when we are counting the queue based on the messages. so for the next time before processing we have to clear the queue.
Here we have multiple queue like slave1, slave2,slave3 and when api will be triggered in the process section we have to clear the queue.
Kindly suggest how we can do this in mule3.
Mule 3 has a generic AMQP connector. It does not support administrative commands from a specific implementation like RabbitMQ, so you can't use the connector.
You could use RabbitMQ REST API and call it using the HTTP Request connector. See this previous answer to see how to delete queues with Curl, then implement the same request with HTTP Request: https://stackoverflow.com/a/29148299/721855
This issue is related to 2 earlier questions:
How to implement HTTP request/reply when the response comes from a rabbitMQ reply queue using Spring Integration DSL?
How do I find the connection information of a RabbitMQ server that is bound to a SCDF stream deployed on Tanzu (Pivotal/PCF) environment?
As you can see the update for the question 2 above, we can receive the correct response back from the rabbit sink. However, it only works half of the time alternated as round-robin way (success-timeout-success-timeout-...). The outside http app was implemented with Spring Integration showed in question 1 - sending the request to the request rabbit source queue and receiving the response from the response rabbit sink queue. This only happened in PCF environment after we deployed both the outside http app and created the stream (see following POC stream) there. However, it's working locally all the time (NOT alternately). Did we miss anything? Not sure what's the culprit in PCF. Thanks.
rabbitSource: rabbit --queues=rabbitSource | my-processor | rabbitSink: rabbit --routing-key=pocStream.rabbitSink.pocStream
Sounds like you have several instances of your stream in that PCF environment. This way there are more then one (round-robin feels like two) subscribers to the same RabbitMQ queue. Where only one consumer must be for that queue since only initiator of the request waits for reply, but odd (or even) replies go to different consumer of the same queue. I don't place it as an answer, just because it is the best guess what is going on since you don't see a problem locally.
Please, investigate your PCF environment and how does it scale instances for your stream. There also might be some option of SCDF which does scaling for us.
I am trying to create a Subscriber in my Spring Boot Application. My objective is that the publisher will send multiple messages to a topic and I have to get those message and process them .I noticed that the "handleMessage" of both Paho and Apache ActiveMq will process 1 message at a time. Is it possible to make it concurrent??
I have tried the following
Replaced Paho with ActiveMq
Provided concurrency in my listenercontainer
Provided prefetch in my subscribe URL
Please let me know if there is any way to make my MQTT subscriber to take multiple messages concurrently.
Thank You
If you supply your own thread pool you can have the handleMessage method pass the incoming message off to the threadpool to process and then pass the next message off to the pool.
How can manual retry work in RabbitMQ after a message has been put onto dead letter queue?
Does RabbitMQ provide an user interface through which you can do this? I assume here that RabbitMQ console does not provide you this capability.
The Rabbit MQ management interface would let you do this crudely, you can go into the deadletter queue, 'get' the message then copy the content. Go to the queue you want to retry the message on and 'publish' it directly to that queue.
Alternatively, you can enable the shovel plugin which allows you to move messages from one queue to another. The RabbitMQ Management plugin directly contains instructions on how to do this.
You can write a consumer / producer using a number of various client libraries. For python a popular library is pika (https://pypi.python.org/pypi/pika).
The script can consume all the messages in a queue, then publish them to another queue.
Right now we have a queuing system(activemq) which is storing the messages. And we have written a separate java application that will read the queue and then trigger a exe to do some processing. But we want to do away with this extra application that is linking our activemq and exe. So i want to know whether any queuing system houses a code which will help me run the exe without any extra code written by me.
Any inputs regarding which queuing system can get me this done will be greatly appreciated.
This isn't really how Message Brokers work. You could however embed a broker inside your own application or create a broker plugin to do something. In the end though the best way to do this is to create your own client that can implement your business logic and let the Message Broker do what it was designed to do, route message traffic.
If you want more of a 'push' solution rather than a producer-consumer solution (which sounds to me like you are) you could look into the use of WebSockets. That would be another way of dealing with messages.
As others have said it doesn't look like using a message broker is the solution you want if you don't want to have some additional middleware to provide asynchronous communication.
So you just need something to launch an EXE on message arrival?
Message Queuing Triggers
Just some additional information for you... Triggered applications are natively supported in IBM Websphere MQ via a Trigger Monitor application that runs as a service (in windows implementations) or a daemon (in UNixish implementations).
When a message arrives in a queue, the MQ software will generate another message ("Trigger" message and send it to the "Trigger" queue, which is being monitored by the Trigger Manager app. The app then starts the required application.
So your implementation of an "app to start an app" is not odd ball or strange at all.. IBM do it in their implementation. I see nothing wrong with your implementation and if you can integrate it tighter to activeMQ then you are on a winner.
What about IBM MQ's triggering feature ?
WebSphere MQ provides a feature that enables an application or channel to be started automatically when there are messages available to retrieve from a queue. A message is put to a queue defined as triggered. If a series of conditions are met, the queue manager sends a trigger message to an initiation queue.
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q026940_.htm