How to check if a storage queue contains a specific message - azure-storage-queues

I'm writing an integration test for an azure function triggered by a storage queue, and I'd like to be able to check if the function has successfully processed the message or moved the message to the poison queue.
Is there any way to search a queue for a specific message, without dequeuing the message?
My approach was to retrieve the messageId and popReceipt of the sent message, and then try to update the message, throwing an exception if not found.
public async Task<bool> IsMessageInQueue(string messageId, string popReceipt, string queueName)
{
try
{
var client = new QueueClient(_storageConnectionString, queueName);
_ = client.UpdateMessageAsync(messageId, popReceipt);
return true; //exists in queue
}
catch (Exception)
{
return false; //doesn't exist in queue
}
}
And then
var sendMessageResponse = await client.SendMessageAsync(queueMessage);
var messageId = sendMessageResponse.Value.MessageId;
var popReceipt = sendMessageResponse.Value.PopReceipt;
var isProcessing = IsMessageInQueue(messageId, popReceipt, "processing");
var isPoisoned = IsMessageInQueue(messageId, popReceipt, "processing-poison");
isProcessing does return true if the message hasn't yet been picked up by the function and is still in "processing", but the problem is that the messageId changes when the message is moved to the poison queue, so isPoisoned will always return false
Update:
Thanks to #gaurav-mantri's suggestion, i updated my method to use PeekMessagesAsync and added my answer below:

Is there any way to search a queue for a specific message, without
dequeuing the message?
It is only possible if the number of messages in your queue is less than 32. 32 is the maximum number of messages you can peek (or dequeue) at a time.
If the number of messages are less than 32, you can use QueueClient.PeekMessagesAsync and compare the message id of your messages with the messages returned. If you find a matching message id, that would mean the message exists in the queue.

Thanks to #gaurav-mantri's suggestion, i updated my method to use PeekMessagesAsync
public async Task<bool> IsMessageInQueue(BinaryData body, string queueName)
{
var client = new QueueClient(_storageConnectionString, queueName);
var messages = await client.PeekMessagesAsync();
return messages.Value.Any(x => x.Body.ToString().Equals(body.ToString()));
}
And this is the usage...
var queueMessage = JsonConvert.SerializeObject("hi");
var sendMessageResponse = await client.SendMessageAsync(queueMessage);
var isProcessing = IsMessageInQueue(new BinaryData(queueMessage), "processing");
var isPoisoned = IsMessageInQueue(new BinaryData(queueMessage), "processing-poison");

Related

cant send message to specific user. connection.on not being reached

I cant seem to reach the client from the hub. All I have is a .on on the client side and I am just trying to reach a break point but its never being reached. I have others on the client side I can reach without issue. I am not receiving any errors and not seeing any script issues in the dev tools console. All I want to do is fire off an action to a specific user. I have tried using both .Client and .User with the same result
On my client I have
connection.on("SendRequest", function (requestmessage) {
var whatever = requestmessage;
});
then in the hub
public async Task RequestPrivateChat(string UserListJS)
{
var ConnectionID = "";
MyUser user = new MyUser();
string message = "This is my message";
dynamic UserList = JsonConvert.DeserializeObject(UserListJS);
foreach (string item in UserList)
{
//I get the user okay then use the user.id below
user = _db.Users.Where(x => x.UserName == item).FirstOrDefault();
//I get the connection ID okay
ConnectionID = _connections.GetConnections(item).First();
//Both of these are reached but the client is never reached out to. Both Connection ID and user.Id are populated correctly
await Clients.Client(ConnectionID).SendAsync("SendRequest", message);
await Clients.User(user.Id.ToString()).SendAsync("SendRequest", message);
}
}
This for sending messages to all clients works fine
Hub
await Clients.All.SendAsync("ReceiveMessage", name, message);
Client
connection.on("ReceiveMessage", function (user, message) {
var msg = message.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
var encodedMsg = user + " says " + msg;
var li = document.createElement("li");
li.textContent = encodedMsg;
document.getElementById("messagesList").appendChild(li);
});
UPDATE
I have tried moving this outside of visual studio into IIS just in case this was and IDE issue with same result. I also tried the following
await Clients.User("username").SendAsync("SendChatRequest", message);
and it still didn't work but oddly enough I realized when I first go to the hub
connection.on("SendRequest", function (requestmessage)
is being hit without ever being called. I have confirmed there are only 2 references to send request, one in the hub and one on the client. I am sure there is something I am missing.
Since your await this.Clients.All.SendAsync("SendRequest", message); works, you not passing correctly the connectionId to target the user.
What I suggest is:
Get the connection Id from the context like:
await Clients.Client(this.Context.ConnectionId).SendAsync("SendRequest", message);
Get the connection Id from the caller:
await Clients.Caller(this.Context.ConnectionId).SendAsync("SendRequest", message);
Add the client to specific group and call the method for the group (assuming you added the connection in to a group):
await Clients.Group("GroupName").SendAsync("SendRequest", message);

Getting RabbitMq to behave as I want (dead letter and re-queue on errors)

I have a simple rabbit set up that is currently doing what I want...
It publishes messages based on the type of message. Each type gets its own queue.
When messages are published they sit on the queue even if there is no consumer to consume them (sit there forever if no consumer arrives).
When a consumer is there (there is only one!) it eats the messages.
If for some reason it cannot handle a message (eg it gets a sub message before the parent has arrived) it nacks the message back onto the queue.
If it sees the same message six times it nacks the message.
This all works, but currently after six attempts it drops the message.
What I would like is for the message to pass to a 'dead letter queue' and after some time (say 5 mins) re-queue that message at the end of the particular queue it came from.
I am definitely cargo cult programing and I don't quite understand all the exchange/queue/binding/routing keys and other arcania involved... hand holding is appreciated!
public void PublishEntity<T>(T message) where T : class, ISendable
{
logger.Info($"publishing {message.UniqueId}");
var factory = new ConnectionFactory
{
HostName = appSettings.RabbitHostName,
UserName = appSettings.RabbitUsername,
Password = appSettings.RabbitPassword
};
try
{
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
Console.WriteLine($"Setting up queues for: {typeof(T).Name}");
channel.QueueDeclare($"App_{typeof(T).Name}",
true,
false,
false,
null);
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.TxSelect();
var properties = channel.CreateBasicProperties();
properties.Persistent = true;
properties.Headers = new Dictionary<string, object>
{
{ "Id", Guid.NewGuid().ToString() }
};
channel.BasicPublish("",
$"App_{typeof(T).Name}",
properties,
body);
Data.MarkAsSent(message);
channel.TxCommit();
}
}
}
ISendable just make sure the message has some properties used in Data.MarkAsSent(message); to mark in a database where we have got too.
The receiver has a similar lump of code to handle each type. As I say this is working.
What do I need to do to add the dead letter queue things?
My attempt like so created the dead letter queues, but nothing ever moves to them.
public void PublishEntity<T>(T message) where T : class, ISendable
{
logger.Info($"publishing {message.UniqueId}");
var factory = new ConnectionFactory
{
HostName = appSettings.RabbitHostName,
UserName = appSettings.RabbitUsername,
Password = appSettings.RabbitPassword
};
try
{
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
Console.WriteLine($"Setting up queues for: {typeof(T).Name}");
channel.ExchangeDeclare("App.Dead.Letter", "direct", true);
var args = new Dictionary<string, object>
{
{ "x-dead-letter-exchange", "App.Dead.Letter" },
{
"x-dead-letter-routing-key", $"DLQ.App_{typeof(T).Name}"
}
};
channel.QueueDeclare($"App_{typeof(T).Name}",
true,
false,
false,
args);
channel.QueueDeclare($"DLQ.App_{typeof(T).Name}",
true,
false,
false,
null);
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.TxSelect();
var properties = channel.CreateBasicProperties();
properties.Persistent = true;
properties.Headers = new Dictionary<string, object>
{
{ "Id", Guid.NewGuid().ToString() }
};
channel.BasicPublish("",
$"App_{typeof(T).Name}",
properties,
body);
Data.MarkAsSent(message);
channel.TxCommit();
}
}
}
In my reciever I have this magic
catch (Exception ex)
{
var attemptsToHandle = MarkFailedToHandleMessage(logId, ex);
if (attemptsToHandle > 5)
{
//If we have seen this message many times then don't re-que.
channel.BasicNack(ea.DeliveryTag, false, false);
return;
}
// re-que so we can re-try later.
channel.BasicNack(ea.DeliveryTag, false, true);
return;
}
Phew... lot of code. Thanks if you have made it this far....
I am asking what are the glaring issues in my code to make things fall to the dead letter queue.
and what extra do I need to add so that things in the dlq will bounce back to the main queue after some time.
Additionally this sets up a dlq for each types queue... is this required or should there be a single queue to hold the errored messages?
So I think I got this working as I intend. Hard to test all this stuff though!
public void PublishEntity<T>(T message) where T : class, ISendable
{
logger.Info($"publishing {message.UniqueId}");
var factory = new ConnectionFactory
{
HostName = appSettings.RabbitHostName,
UserName = appSettings.RabbitUsername,
Password = appSettings.RabbitPassword
};
try
{
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
Console.WriteLine($"Setting up queues for: {typeof(T).Name}");
// Declair dead letter queue for this type
channel.ExchangeDeclare("App.Dead.Letter", "direct", true);
var queueArgs = new Dictionary<string, object>
{
{ "x-dead-letter-exchange", "App" },
{
"x-dead-letter-routing-key", $"App_{typeof(T).Name}"
}
,{ "x-message-ttl", 30000 }
};
channel.QueueDeclare($"DLQ.App_{typeof(T).Name}",
true,
false,
false,
queueArgs);
channel.QueueBind($"DLQ.App_{typeof(T).Name}", "App.Dead.Letter", $"DLQ.App_{typeof(T).Name}", null);
// declair queue for this type
channel.ExchangeDeclare("App", "direct", true);
var args = new Dictionary<string, object>
{
{ "x-dead-letter-exchange", "App.Dead.Letter" },
{
"x-dead-letter-routing-key", $"DLQ.App_{typeof(T).Name}"
}
};
channel.QueueDeclare($"App_{typeof(T).Name}",
true,
false,
false,
args);
channel.QueueBind($"App_{typeof(T).Name}", "App", $"App_{typeof(T).Name}", null);
I added and exchange for my main queues to live in and actually bound the queues to the exchanges. I still don't know why I need to do this as it was working without this extra complexity.. I guess some magic was hiding it from me before?

NServiceBus MSMQ, How to Stop message transfer to Error Queue

I am using NServiceBus for a proof of concept. I am using it with MSMQ. I Have a Web Application which send a message. This message will be handled by a subscriber or message handler. When i shutdown the message handler, message will be send to Error Queue after few retries.
I don't want to send it to error queue, i want to keep message in Input queue as long as Handler is again online and process this message automatically from input queue. If message can not be processed from handler due to any error, it must transfer message to error queue.
Here is my configuration in MVC application Global.asax
private void ConfigureServiceBus()
{
var endpointConfiguration = new EndpointConfiguration("Samples.Mvc.Endpoint");
endpointConfiguration.UseSerialization<JsonSerializer>();
endpointConfiguration.UsePersistence<InMemoryPersistence>();
endpointConfiguration.UseTransport<MsmqTransport>();
endpointConfiguration.SendFailedMessagesTo("error");
endpointConfiguration.AuditProcessedMessagesTo("audit");
var recoverability = endpointConfiguration.Recoverability();
recoverability.Delayed(
delayed =>
{
delayed.NumberOfRetries(2);
delayed.TimeIncrease(TimeSpan.FromSeconds(2));
});
endpointConfiguration.EnableInstallers();
endpoint = Endpoint.Start(endpointConfiguration).GetAwaiter().GetResult();
//var endpointInstance = Endpoint.Start(endpointConfiguration).ConfigureAwait(false);
var mvcContainerBuilder = new ContainerBuilder();
mvcContainerBuilder.RegisterInstance(endpoint);
// Register MVC controllers.
mvcContainerBuilder.RegisterControllers(typeof(MvcApplication).Assembly);
var mvcContainer = mvcContainerBuilder.Build();
DependencyResolver.SetResolver(new AutofacDependencyResolver(mvcContainer));
}
This is the part in Controller where i send message to endpoint.
[HttpPost]
public async Task<ActionResult> Contact(ContactModel c)
{
if (ModelState.IsValid)
{
try
{
var message = new ContactMessage()
{
Comment = c.Comment,
Email = c.Email,
FirstName = c.FirstName,
LastName = c.LastName,
TransectionId = Guid.NewGuid()
};
await endpoint.Send("Samples.Mvc.Endpoint", message).ConfigureAwait(false);
return View("Success");
}
catch (Exception ex)
{
return View("Error");
}
}
return View();
}
My goal is to keep unhandled messages in input queue as long as message handler is not back on work. currently if message handler is down, it send messages after retries to Error Queue.
I am using NServiceBus 6.x
thanks in advance

How to consume an object from azure service bus topic subscription

I got this error upon receving an object from a subscription in azure service bus.
An exception of type 'System.Runtime.Serialization.SerializationException' occurred in System.Runtime.Serialization.dll but was not handled in user code
I've tried some deserialization code but nothing works.
This is how I send a message. Please tell me how to receive it.
public void SendMessage()
{
BrokeredMessage message = new BrokeredMessage(new TestMessage() {
MsgNumber = 1, MsgContent = "testing message" }, new DataContractSerializer(typeof(TestMessage)));
// Send message to the topic
TopicClient topicClient = TopicClient.CreateFromConnectionString(cn, topicNamespace);
topicClient.Send(message);
}
public string ReceiveMessage(){
//??????
}
To receive a single message, you need to get the SubscriptionClient :
public void ReceiveMessage(string connectionString, string topicPath, string subscriptionName)
{
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, topicPath, subscriptionName);
var brokeredMessage = subscriptionClient.Receive();
var message = brokeredMessage.GetBody<TestMessage>();
}

Custom Client MessageInspector recording requests but not responses

I have a Custom ClientMessageInspector that records requests but not replies to my service.
The code is:
namespace MessageListener.Instrumentation
{
public class MessageInspector : IClientMessageInspector
{
private Message TraceMessage(MessageBuffer buffer)
{
// Must use a buffer rather than the original message, because the Message's body can be processed only once.
Message msg = buffer.CreateMessage();
using (RREM_GilbaneEntities3 entities3 = new RREM_GilbaneEntities3())
{
SOAPMessage soapMessages = new SOAPMessage
{
SOAPMessage1 = msg.ToString(),
created = DateTime.Now,
source = "Interface12",
sourceIP = "Interface12"
};
entities3.SOAPMessages.Add(soapMessages);
entities3.SaveChanges();
}
//Return copy of origonal message with unalterd State
return buffer.CreateMessage();
}
public void AfterReceiveReply(ref System.ServiceModel.Channels.Message reply, object correlationState)
{
reply = TraceMessage(reply.CreateBufferedCopy(int.MaxValue));
}
public object BeforeSendRequest(ref System.ServiceModel.Channels.Message request, System.ServiceModel.IClientChannel channel)
{
request = TraceMessage(request.CreateBufferedCopy(int.MaxValue));
return null;
}
}
}
What seems to be happening is both AfterRecievReply and BeforeSendRequest are being called. In AfterRecieveReply before I call TraceMessage, I can see the whole reply. Inside TraceMessage, when I do:
// Must use a buffer rather than the original message, because the Message's body can be processed only once.
Message msg = buffer.CreateMessage();
it turns the reply into junk:
msg {<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header />
<soap:Body>... stream ...</soap:Body>
</soap:Envelope>}
What's going on?
The reply isn't a junk message - it's just when you call ToString on it that it doesn't show the body of the message. Remember that a message can only be consumed once; once its body is read, it cannot be read again. Since many places (including the watch window of debuggers) will call ToString on an object, this method is implemented in a way that if it doesn't know for sure that a message body can be read multiple times, then it won't, which seems to be your case. If you want to really write out the message, try using this code:
public string MessageToString(Message message) {
using (MemoryStream ms = new MemoryStream()) {
XmlWriterSettings ws = new XmlWriterSettings();
ws.Encoding = new UTF8Encoding(false);
using (XmlWriter w = XmlWriter.Create(ms)) {
message.WriteMessage(w);
w.Flush();
return ws.Encoding.GetString(ms.ToArray());
}
}
}