Need Help to understand this..
My application is listening to IBM MQ (On new Message).This MQ is subscribed to a topic., when messages are loaded to topic, my application processes them.
Having said that., attaching the logs, here mule event ids are not unique for each message.
Also, im logging correlationId as my job guid to track one end to end transaction in mule.
But none of the Ids are unique.
Also, im guessing the MessageID is being logged as eventID of mule - Correct me if wrong.
Note: I have also set 'disable Message ID' to true in my IBM MQ listener.
I just want to know why event ids or corelation id are not unique and to track one complete transaction in mule., what can be used?
Edited:
Logs as described - event Id of 2 different applications.
INFO 2023-02-07 07:23:28,506 [[MuleRuntime].uber.65573: [app-name].app-name-1-Flow.CPU_LITE #19dd2f11] [processor: app-name-1-Flow/processors/0; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
INFO 2023-02-07 07:23:28,119 [[MuleRuntime].uber.65571: [app-name].app-name-1-Flow.CPU_LITE #19dd2f11] [processor: app-name-1-Flow/processors/0; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
INFO 2023-02-07 07:21:34,373 [[MuleRuntime].uber.91422: [app-name].app-name-Flow.CPU_LITE #60d70fe6] [processor: app-name-Flow/processors/0; event: ID:414d51205150415a303555582020202005aa0563048ba823] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
INFO 2023-02-07 07:21:34,355 [[MuleRuntime].uber.91422: [app-name].app-name-Flow.CPU_LITE #60d70fe6] [processor: app-name-Flow/processors/0; event: ID:414d51205150415a303555582020202005aa0563048ba823] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
Logs for jobGuid (set as correlation ID) is same as eventID
INFO 2023-02-07 07:23:28,123 [[MuleRuntime].uber.65572: [app-name].app-name-1-Flow.CPU_INTENSIVE #10f2a539] [processor: flow-name-SubFlow/processors/0; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","txnGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"ProcessedSub","sourceEpoc":"1675772608120","now":"1675772608122"}}
INFO 2023-02-07 07:23:28,122 [[MuleRuntime].uber.65572: [app-name].app-name-1-Flow.CPU_INTENSIVE #10f2a539] [processor: app-name-1-Flow/processors/3; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","txnGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"Received","sourceEpoc":"1675772608120","now":"1675772608121"}}
INFO 2023-02-07 07:21:34,656 [[MuleRuntime].uber.91422: [app-name].app-name-2-Flow.CPU_INTENSIVE #1ae0a3cf] [processor: app-name-2-Flow/processors/4/route/0/processors/2; event: ID:414d51205150415a303555582020202005aa0563048ba823] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","txnGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"Received","sourceEpoc":"1675772494374","now":"1675772494656"}}
INFO 2023-02-07 07:21:34,653 [[MuleRuntime].uber.91419: [app-name].app-name-2-Flow.CPU_INTENSIVE #1ae0a3cf] [processor: app-name-2-Flow/processors/4/route/0/processors/2; event: ID:414d51205150415a303555582020202005aa0563048ba823] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","txnGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"Received","sourceEpoc":"1675772494355","now":"1675772494653"}}
Event Ids -also called correlation ids- are usually unique when generated by Mule automatically. When overridden with the message id from a queue broker, like IBM MQ, it is up to the message generator to send unique ids. In the log snippets you shared the event id looks to be a sequence of bytes in hexadecimal. That's usual for IBM MQ message ids. Default Mule event ids are formatted GUIDs.
Related
I am trying to make an SSE Spring application, using Webflux. According to the documentation, the message is not sent to the sink if there is no subscriber. In my use case, I would like that the subscriber would receive the last message when calling for subscription. I have found that Sink can be configured in following way:
Sinks.many().replay().latest();
And when I have both publisher and subscriber, and the next subscriber calls for subscription, he receives the last sent message, which is great. However if I don't have any subscribers, publisher sends the message and then first subscriber comes in, it receives none. Which is just as documentation above says actually, but I am thinking how to solve that issue to meet my needs. As a workaround I did something like this:
if (shareSinks.currentSubscriberCount() == 0) {
shareSinks.asFlux().subscribe();
}
shareSinks.tryEmitNext(shareDTO);
But subscribing the publisher to its own subscription doesn't sound like a clean way to do this...
This is a matter of hot and cold publishers. Currently, your publisher (Sinks.many().replay().latest()) is a cold publisher. Events that are being emitted while there is no subscriber, will just vanish.
What you need is a so called hot publisher. Hot publishers cache the events and a new subscriber will receive all previously cached events.
This will do the trick:
final Sinks.Many<String> shareSinks = Sinks.many()
.replay()
.all(); // or .limit(10); to keep only the last 10 emissions
final Flux<String> hotPublisher = shareSinks.asFlux()
.cache(); // .cache() turns the cold flux into a
// hot flux
shareSinks.tryEmitNext("1");
shareSinks.tryEmitNext("2");
shareSinks.tryEmitNext("3");
shareSinks.tryEmitNext("4");
hotPublisher.subscribe(message -> System.out.println("received: " + message));
The console print out would be:
received: 1
received: 2
received: 3
received: 4
The Reactor docu also has a chapter on hot vs. cold.
When using StreamMessageListenerContainer a subscription for a consumer group can be created by calling:
receive(consumer, readOffset, streamListener)
Is there a way to configure the container/subscription so that it will always attempt to re-process any PENDING messages before moving on to polling for new messages?
The goal would be to keep retrying any message that wasn't acknowledged until it succeeds, to ensure that the stream of events is always processed in exactly the order it was produced.
My understanding is if we specify the readOffset as '>' then on every poll it will use '>' and it will never see any messages from the PENDING list.
If we provide a specific message id, then it can see messages from the PENDING list, but the way the subscription updates the lastMessageId is like this:
pollState.updateReadOffset(raw.getId().getValue());
V record = convertRecord(raw);
listener.onMessage(record);
So even if the listener throws an exception, or just doesn't acknowledge the message id, the lastMessageId in pollState is still updated to this message id and won't be seen again on the next poll.
How do i disable logs for message properties. I need to call Https of a web service and get the results back, but these messages came back with it in the logging. I am not allowed to show these confidential information at all.
Thanks
INFO 2017-05-23 15:48:06,950 [[ManulifeMay23].HTTP_Listener_Configuration.worker.01] org.mule.api.processor.LoggerMessageProcessor:
org.mule.DefaultMuleMessage
{
id=bdf0b820-3ff0-11e7-b37a-005056a37828
payload=java.util.LinkedHashMap
correlationId=bdeac4b0-3ff0-11e7-b37a-005056a37828
correlationGroup=31
correlationSeq=1
encoding=UTF-8
exceptionPayload=<not set>
ESPECIALLY this invocation parts
Message properties:
INVOCATION scoped properties:
bonusAmount=0
bounsCounter=0
counter=1
cxf_operation=}getBonusPoint
cxf_service={}ExternalServerService
jsonVariable={selectFields=[programName, registrationDate, cycleToDateAdjustedBonusAmount, totalEarnedRedeemableBonusAmount], **common={accountId=4040320030000877}}**
lastFourCC=0877
operation=getBonusPoint
payload1=[Ljava.lang.Object;#4ea65f0
programName=
registrationDate=
rootMessage=<<<MuleMessage>>>
totalAmount=0
INBOUND scoped properties:
connection=Keep-Alive
content-type=application/json
date=Tue, 23 May 2017 19:48:06 GMT
http.reason=OK
http.status=200
transfer-encoding=chunked
x-archived-client-ip=10.180.205.250
x-backside-transport=OK OK,OK OK
x-client-ip=10.180.205.250,10.188.196.7
x-global-transaction-id=962389919
x-response-code=200
OUTBOUND scoped properties:
MULE_CORRELATION_GROUP_SIZE=31
MULE_CORRELATION_ID=bdeac4b0-3ff0-11e7-b37a-005056a37828
MULE_CORRELATION_SEQUENCE=1
SESSION scoped properties:`enter code here`
}
This is the default log if you don't declare any value on logger. You may want to customize the value on logger so that this will not be display.
I have an exchange setup amq.topic and in there I have a routing key that is pointing to my key.
I have a route defined with a rest endpoint. I make a call to that endpoint with a body as json {"sample" : "sample"}.
rest("myendpoint")
.post()
.route()
.routeId(ROUTE_ID)
.log(ROUTE_ID + " started.")
.convertBodyTo(String.class)
.log("Receiving request with body=${body}")
.to("rabbitmq://rabbiturl:port/amq.topic?connectionFactory=rabbitmqConnectionFactory&autoDelete=false&routingKey=myroutingkey&declare=false&exchangeType=topic")
.log(ROUTE_ID + " ended.")
.end();
This accomplishes what I want to: the body of my rest call will be sent to the topic exchange of which will go to my queue based on the routing key.
The problem: It also create an auto-generated queue
[ qtp86171426-28] RabbitMQProducer INFO Starting reply manager service RabbitMQReplyManagerTimeoutChecker[amq.topic]
[ qtp86171426-28] ReplyManagerSupport INFO Using temporary queue name: amq.gen-JGF0CKioQ6_MLA9uBNIOOQ
[ qtp86171426-28] CorrelationTimeoutMap INFO in putIfAbsent with key Camel-ID-L-VC-10050-58983-1467158153454-0-3
[nagerTimeoutChecker[amq.topic]] TemporaryQueueReplyHandler INFO in onTimeout with correlationId= Camel-ID-L-VC-10050-58983-1467158153454-0-3
[nagerTimeoutChecker[amq.topic]] ReplyManagerSupport WARN Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-L-VC-10050-58983-1467158153454-0-3] on destination amq.gen-JGF0CKioQ6_MLA9uBNIOOQ. Setting ExchangeTimedOutException on (MessageId: ID-L-VC-10050-58983-1467158153454-0-2 on ExchangeId: ID-L-VC-10050-58983-1467158153454-0-1) and continue routing.
I have tried skipqueuedeclare & skipexchangedeclare and am running out of ideas of why it is generating the queue. Does anyone have any ideas?
Set ExchangePattern to InOnly.
The temporary and reply queue is because the route expects a reply back so creates the temporary queue to wait for a response.
So if your rest endpoint does not return anything set the ExchangePattern to Inonly at the rest call and it should be ok. Alternatively set it at .to().
I am not able to get the reason of exception from the message that is in the DLQ.
These are the steps i have followed:-
Message is sent to a 'Sample' Queue.
The Message Listener throws a Runtime Exception in the onMessage Function.(throw new RuntimeException("Exception Reason Test");)
The message goes to the DLQ.
I am trying to access the Exception reason via two approaches(i pass the DLQ Name and the JMS Message ID in both):-
Spring JMSTemplate browseSelected function
(ActiveMQMessage)message.getStringProperty(ActiveMQMessage.DLQ_DELIVERY_FAILURE_CAUSE_PROPERTY);
JMX QueueViewMBean browse function via the composite data map
Map datamap = (Map)dataMap.get("StringProperties");
datamap.containsKey(ActiveMQMessage.DLQ_DELIVERY_FAILURE_CAUSE_PROPERTY);
In both cases i am getting a null value for the property DLQ_DELIVERY_FAILURE_CAUSE_PROPERTY.
Please help.
Thanks