I'm at consumer end and reading message from a queue. How to get the count of the total messages in a queue (IDestination) and pass number of messages (let's say 100 messages) at a time to the receiver.
Please help to achieve it in c# .Net
One means of achieving this would be to enable the Statistics Broker Plugin and then send control commands to the broker from your C# client to obtain statistics on Queues you are interested in. The follow is a Java based example but you could produce something quite similar using Apache.NMS.ActiveMQ if that is what you are using for your C# client.
Queue replyTo = session.createTemporaryQueue();
MessageConsumer consumer = session.createConsumer(replyTo);
Queue testQueue = session.createQueue("TEST.FOO");
MessageProducer producer = session.createProducer(null);
String queueName = "ActiveMQ.Statistics.Destination." + testQueue.getQueueName()
Queue query = session.createQueue(queueName);
Message msg = session.createMessage();
producer.send(testQueue, msg)
msg.setJMSReplyTo(replyTo);
producer.send(query, msg);
MapMessage reply = (MapMessage) consumer.receive();
assertNotNull(reply);
assertTrue(reply.getMapNames().hasMoreElements());
for (Enumeration e = reply.getMapNames();e.hasMoreElements();) {
String name = e.nextElement().toString();
System.err.println(name + "=" + reply.getObject(name));
}
That would allow you to see data such as:
memoryUsage=0
dequeueCount=0
inflightCount=0
messagesCached=0
averageEnqueueTime=0.0
destinationName=queue://TEST.FOO
size=1
memoryPercentUsage=0
producerCount=0
consumerCount=0
minEnqueueTime=0.0
maxEnqueueTime=0.0
dispatchCount=0
expiredCount=0
enqueueCount=1
memoryLimit=67108864
Related
I'm using RabbitMQ. I've defined a queue with priority, and I can send messages to this queue with some priority value using RMQ GUI, and consumers also get the messages in sorted order, but when I try to send the message from my java code using Stream bridge, I don't know how to specify the priority with the message.
Here's what I have tried :
I have added x-max-priority: 10 to the queue while creating the queue.
Consumer example =
#Bean
public Consumer<Message<String>> testListener() {
return (m) -> {
System.out.println("inside consumer with message : " + m);
System.out.println("headers : " + m.getHeaders());
System.out.println("payload : " + m.getPayload());
};
}
Producer example =
#GET
#Path("test/")
public void test(#Context HttpServletRequest request) {
System.out.println("inside test");
try {
String payload = "hello world";
logger.info("going to send a message : {}", payload);
int priority = 5;
Message<String> message = MessageBuilder.withPayload(payload)
.setHeader("priority", priority)
.build();
boolean res = STREAM_BRIDGE.send("testWriter-out-0", message);
System.out.println(message);
System.out.println(res);
} catch (Exception e) {
logger.error(e);
}
}
The output of the Producer =
-> inside test
-> GenericMessage [payload=hello world, headers={priority=5, id=some_id, timestamp=epoch}]
-> true
The output of the Consumer =
-> inside consumer with message : GenericMessage [payload=hello world, headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=test_exchange, amqp_deliveryTag=1, deliveryAttempt=1, amqp_consumerQueue=test_exchange.ats, amqp_redelivered=false, amqp_receivedRoutingKey=test_exchange, amqp_timestamp=date_time, amqp_messageId=some_id, id=some_id, amqp_consumerTag=some_tag, sourceData=(Body:'hello world' MessageProperties [headers={}, timestamp=date_time, messageId=some_id, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=test_exchange, receivedRoutingKey=test_exchange, deliveryTag=1, consumerTag=some_tag, consumerQueue=test_exchange.ats]), contentType=application/json, timestamp=epoch}]
-> headers : {amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=test_exchange, amqp_deliveryTag=1, deliveryAttempt=1, amqp_consumerQueue=test_exchange.ats, amqp_redelivered=false, amqp_receivedRoutingKey=test_exchange, amqp_timestamp=date_time, amqp_messageId=some_id, id=some_id, amqp_consumerTag=tag, sourceData=(Body:'hello world' MessageProperties [headers={}, timestamp=date_time, messageId=some_id, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=test_exchange, receivedRoutingKey=test_exchange, deliveryTag=1, consumerTag=tag, consumerQueue=test_exchange.ats]), contentType=application/json, timestamp=epoch}
-> payload : hello world
So the message goes to RMQ and the consumer also gets the message, but on RMQ GUI when I perform Get-message operation on the Queue, I get this result =>
Message 1
The server reported 0 messages remaining.
Exchange test_exchange
Routing Key test_exchange
Redelivered ○
Properties
timestamp: timestamp
message_id: some_id
priority: 0
delivery_mode: 2
headers:
content_type: application/json
Payload hello world
11 bytes
Encoding: string
As we can see in the above result, priority is set to 0 by RMQ (and hence in the Consumer, I get the messages in the FIFO manner, not in a priority-based manner) and inside headers : only one header is present "content_type: application/json", so I think the priority is not a part of the header but is a part of properties, then how to set message properties using StreamBridge?
To conclude, I am trying to figure out how to set the priority of a message dynamically while sending it using StreamBridge, any help would be appreciated, thanks in advance !
Please, consider to use the latest Spring Cloud Stream: https://spring.io/projects/spring-cloud-stream#learn.
Apparently your spring-cloud-starter-stream-rabbit = 3.0.3.RELEASE is old enough to suffer from the issue https://github.com/spring-cloud/spring-cloud-stream/issues/1931.
Have just tested with the latest one and I got the proper priority property on the message posted into RabbitMQ queue by the mentioned StreamBridge.
I have a RabbitMQ server (v.3.8.2) with a simple exchange fanout-queue binding running, with several producers and one consumer. The average delivery/ack rate is quite low, about 6 msg/s.
The queue is created at runtime by producers with the x-message-ttl parameter set at 900000 (15 minutes).
In very specific conditions (e.g. rare error situation), messages are rejected by the consumer. These messages then are shown in the unacked counter on the RabbitMQ admin web page indefinitely. They never expire or get discarded event after they timeout.
There are no specific per-message overrides in ttl parameters.
I do not need any dead letter processing as these particular messages do not require processing high reliabilty, and I can afford to lose some of them every now and then under those specific error conditions.
Exchange parameters:
name: poll
type: fanout
features: durable=true
bound queue: poll
routing key: poll
Queue parameters:
name: poll
features: x-message-ttl=900000 durable=true
For instance, this is what I am currently seeing in the RabbitMQ server queue admin page:
As you can see, there are 12 rejected/unack'ed messages in the queue, and they have been living there for more than a week now.
How can I have the nacked messages expire as per the ttl parameter?
Am I missing some pieces of configuration?
Here is an extract from the consumer source code:
// this code is executed during setup
...
consumer = new EventingBasicConsumer(channel);
consumer.Received += (sender, e) =>
{
// Retrieve retry count & death list if present
List<object> DeathList = ((e?.BasicProperties?.Headers != null) && e.BasicProperties.Headers.TryGetValue("x-death", out object obj)) ? obj as List<object> : null;
int count = ((DeathList != null) &&
(DeathList.Count > 0) &&
(DeathList[0] is Dictionary<string, object> values) &&
values.TryGetValue("count", out obj)
) ? Convert.ToInt32(obj) : 0;
// call actual event method
switch (OnRequestReceived(e.Body, count, DeathList))
{
default:
channel.BasicAck(e.DeliveryTag, false);
break;
case OnReceivedResult.Reject:
channel.BasicReject(e.DeliveryTag, false);
break;
case OnReceivedResult.Requeue:
channel.BasicReject(e.DeliveryTag, true);
break;
}
};
...
// this is the actual "OnReceived" method
static OnReceivedResult OnRequestReceived(byte[] payload, int count, List<object> DeathList)
{
OnReceivedResult retval = OnReceivedResult.Ack; // success by default
try
{
object request = MessagePackSerializer.Typeless.Deserialize(payload);
if (request is PollRequestContainer prc)
{
Log.Out(
Level.Info,
LogFamilies.PollManager,
log_method,
null,
"RequestPoll message received did={0} type=={1} items#={2}", prc.DeviceId, prc.Type, prc.Items == null ? 0 : prc.Items.Length
);
if (!RequestManager.RequestPoll(prc.DeviceId, prc.Type, prc.Items)) retval = OnReceivedResult.Reject;
}
else if (request is PollUpdateContainer puc)
{
Log.Out(Level.Info, LogFamilies.PollManager, log_method, null, "RequestUpdates message received dids#={0} type=={1}", puc.DeviceIds.Length, puc.Type);
if (!RequestManager.RequestUpdates(puc.DeviceIds, puc.Type)) retval = OnReceivedResult.Reject;
}
else Log.Out(Level.Error, LogFamilies.PollManager, log_method, null, "Message payload deserialization error length={0} count={1}", payload.Length, count);
}
catch (Exception e)
{
Log.Out(Level.Error, LogFamilies.PollManager, log_method, null, e, "Exception dequeueing message. Payload length={0} count={1}", payload.Length, count);
}
// message is rejected only if RequestUpdates() or RequestPoll() return false
// message is always acked if an exception occurs within the try-catch or if a deserialization type check error occurs
return retval;
}
This state occurs when the Consumer does not ack or reject both after receiving the message.
In the unacked state, the message does not expire.
After receiving the message, you must ack or reject it.
This issue isn't a problem that doesn't expire, problem is you don't ack or reject the message.
x-message-ttl=900000 means how long the message stays in the queue without being delivered to the consumer.
In your situation, your message is already delivered to the consumer and it needs to be acked/rejected.
A follow up to this:
one SCDF source, 2 processors but only 1 processes each item
The 2 processors (del-1 and del-2) in the picture are receiving the same data within milliseconds of each other. I'm trying to rig this so del-2 never receives the same thing as del-1 and vice versa. So obviously I've got something configured incorrectly but I'm not sure where.
My processor has the following application.properties
spring.application.name=${vcap.application.name:sample-processor}
info.app.name=#project.artifactId#
info.app.description=#project.description#
info.app.version=#project.version#
management.endpoints.web.exposure.include=health,info,bindings
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration
spring.cloud.stream.bindings.input.group=input
Is "spring.cloud.stream.bindings.input.group" specified correctly?
Here's the processor code:
#Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)
public Object transform(String inputStr) throws InterruptedException{
ApplicationLog log = new ApplicationLog(this, "timerMessageSource");
String message = " I AM [" + inputStr + "] AND I HAVE BEEN PROCESSED!!!!!!!";
log.info("SampleProcessor.transform() incoming inputStr="+inputStr);
return message;
}
Is the #Transformer annotation the proper way to link this bit of code with "spring.cloud.stream.bindings.input.group" from application.properties? Are there any other annotations necessary?
Here's my source:
private String format = "EEEEE dd MMMMM yyyy HH:mm:ss.SSSZ";
#Bean
#InboundChannelAdapter(value = Source.OUTPUT, poller = #Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
public MessageSource<String> timerMessageSource() {
ApplicationLog log = new ApplicationLog(this, "timerMessageSource");
String message = new SimpleDateFormat(format).format(new Date());
log.info("SampleSource.timeMessageSource() message=["+message+"]");
return () -> new GenericMessage<>(new SimpleDateFormat(format).format(new Date()));
}
I'm confused about the "value = Source.OUTPUT". Does this mean my processor needs to be named differently?
Is the inclusion of #Poller causing me a problem somehow?
This is how I define the 2 processor streams (del-1 and del-2) in SCDF shell:
stream create del-1 --definition ":split > processor-that-does-everything-sleeps5 --spring.cloud.stream.bindings.applicationMetrics.destination=metrics > :merge"
stream create del-2 --definition ":split > processor-that-does-everything-sleeps5 --spring.cloud.stream.bindings.applicationMetrics.destination=metrics > :merge"
Do I need to do anything differently there?
All of this is running in Docker/K8s.
RabbitMQ is given by bitnami/rabbitmq:3.7.2-r1 and is configured with the following props:
RABBITMQ_USERNAME: user
RABBITMQ_PASSWORD <redacted>:
RABBITMQ_ERL_COOKIE <redacted>:
RABBITMQ_NODE_PORT_NUMBER: 5672
RABBITMQ_NODE_TYPE: stats
RABBITMQ_NODE_NAME: rabbit#localhost
RABBITMQ_CLUSTER_NODE_NAME:
RABBITMQ_DEFAULT_VHOST: /
RABBITMQ_MANAGER_PORT_NUMBER: 15672
RABBITMQ_DISK_FREE_LIMIT: "6GiB"
Are any other environment variables necessary?
I migrated app from vb6 to vb.net, every thing is working fine except getting the message from the MQ server, when MQGet called, I get 2033 error (no Message)
also attached the capture message that sent to MQ from working vb6 and from not working vb.net, please help?enter image description here
'***********************************
'Send(MQPUT) to MQSeries and CICS
'***********************************
PutMsgOpts = MQPMO_DEFAULT
MsgDesc = MQMD_DEFAULT
MsgDesc.Persistence = MQPER_PERSISTENT
MsgDesc.MsgId = MQMI_NONE.Value
MsgDesc.CorrelId = "AMQ!NEW_SESSION_CORRELID" 'if using MQBridge
MsgDesc.ReplyToQ = gReplyToQ
MsgDesc.ReplyToQMgr = gMQRplyMgrName
MsgDesc.Format_Renamed = MQFMT_STRING
'sPutMsg is composed of 8 byte program name(host/cics)
' plus data desired to pass as string only (dfcommarea)
'************ MAX LENGTH IS 32776 *******************
sPutMsg = gCICSPrgName & gsHost_Msg
If Len(sPutMsg) <> 32768 Then
sPutMsg = sPutMsg & Space(32768 - Len(sPutMsg))
End If
sMsgIdGet.Value = MsgDesc.MsgId
'*******************************************
'MQClose the queue for request sent to host
'*******************************************
'HOST-BYPASS
MQCLOSE(Hconn, Hobj, MQCO_NONE, cC, Reason)
'HOST-BYPASS
'**********************************************************
'MQOpen the queue for receiving the request from the host
'**********************************************************
'set up the queue name
ObjDesc = MQOD_DEFAULT
ObjDesc.ObjectName = gReplyToQ
'Open
'HOST-BYPASS
MQOPEN(Hconn, ObjDesc, MQOO_INPUT_AS_Q_DEF Or MQOO_FAIL_IF_QUIESCING, Hobj, cC, Reason)
'HOST-BYPASS
'Hconn is set by MQseries in the Connect
'**************************************
'Receive (MQGET) from MQSeries and CICS
'**************************************
GetMsgOpts = MQGMO_DEFAULT
GetMsgOpts.Options = MQGMO_WAIT Or MQGMO_CONVERT
GetMsgOpts.WaitInterval = 10000 '20 seconds
MsgDesc = MQMD_DEFAULT
MsgDesc.Format_Renamed = MQFMT_STRING
MsgDesc.Persistence = MQPER_PERSISTENT
MsgDesc.CorrelId = sMsgIdGet.Value 'set correlId with MQGET msgid returned
MsgDesc.MsgId = MQMI_NONE.Value
MsgDesc.ReplyToQ = gReplyToQ
MsgDesc.ReplyToQMgr = gMQRplyMgrName
'** sGetMsg is composed of 8 byte program name(host/cics)
'** plus data desired to pass as string only (dfcommarea)
'************ MAX LENGTH IS 32776 *******************
sGetMsg = New String(" ", 32768)
'Command to receive from to MQSeries and CICS
retryCount = 0
cC = MQCC_OK + 1
Reason = 2033
Do While cC <> MQCC_OK And Reason = 2033
MQGET(Hconn, Hobj, MsgDesc, GetMsgOpts, Len(sGetMsg), sGetMsg, readlen, cC, Reason)
retryCount = retryCount + 1
If retryCount > 1 Then Exit Do
Loop
'************************
'MQClose queue for input
'************************
MQCLOSE(Hconn, Hobj, MQCO_NONE, cC, Reason)
If you get 2033, then you know the queue is there, but the message you are trying to retrieve is not. Use MQ Explorer or plain MQ samples along with runmqsc to check if there are any messages on that queue, and if there are, if their correlation id matches what your program is supplying.
If there are no messages there, then check that your messages have been indeed received by the CICS program your are sending them to, and if that program responded.
A 2033 problem may be your coding problem, but more probably it's something in the configuration and setup that has changed along with your conversion. You may have a differently configured queue manager, with differently configured channels etc. Check the entire chain.
Just spent my first few hours looking at Redis and Redis MQ.
Slowly getting the hang of Redis and was wondering how you could resend a message that is in a dead letter queue?
Also, where are the configuration options which determine how many times a message is retried before it goes into the dead letter queue?
Currently, there's no way to automatically resend messages in the dead letter queue in ServiceStack. However, you can do this manually relatively easily:
To reload messages from the dead letter queue by using:
public class AppHost {
public override Configure(){
// create the hostMq ...
var hostMq = new RedisMqHost( clients, retryCount = 2 );
// with retryCount = 2, 3 total attempts are made. 1st + 2 retries
// before starting hostMq
RecoverDLQMessages<TheMessage>(hostMq);
// add handlers
hostMq.RegisterHandler<TheMessage>( m =>
this.ServiceController.ExecuteMessage( m ) );
// start hostMq
hostMq.Start();
}
}
Which ultimately uses the following to recover (requeue) messages:
private void RecoverDLQMessages<T>( RedisMqHost hostMq )
{
var client = hostMq.CreateMessageQueueClient();
var errorQueue = QueueNames<T>.Dlq;
log.Info( "Recovering Dead Messages from: {0}", errorQueue );
var recovered = 0;
byte[] msgBytes;
while( (msgBytes = client.Get( errorQueue, TimeSpan.FromSeconds(1) )) != null )
{
var msg = msgBytes.ToMessage<T>();
msg.RetryAttempts = 0;
client.Publish( msg );
recovered++;
}
log.Info( "Recovered {0} from {1}", recovered, errorQueue );
}
Note
At the time of this writing, there's a possibility of ServiceStack losing messages. Please See Issue 229 Here, so, don't kill the process while it's moving messages from the DLQ (dead letter queue) back to the input queue. Under the hood, ServiceStack is POPing messages from Redis.