RabbitMQ-- selectively retrieving messages from a queue - rabbitmq

I'm new to RabbitMQ and was wondering of a good approach to this problem I'm mulling over. I want to create a service that subscribes to a queue and only pulls messages that meet a specific criteria; for instance, if a specific subject header is in the message.
I'm still learning about RabbitMQ, and was looking for tips on how to approach this. My questions include: how can the consumer pull only specific messages from the queue? How can the producer set a subject header in the message (if that's even the right term?)

RabbitMQ is perfect for this situation. You have a number of options to do what you want. I suggest reading the documentation to get a better understanding. I would suggest that you use a topic or direct exchange. Topic is more flexible. It goes like this.
Producer code connects to the RabbitMQ Broker and creates and Exchange with a specific name.
Producer publishes to exchange. Each message published will be published with a routing key.
Consumer connects to RabbitMQ broker.
Consumer creates Queue
Consumer binds Queue to the exchange, the same exchange defined in the producer. The binding also includes the routing keys for each message require for this particular consumer.
Lets say you were publishing log messages. The routing key might be something like "log.info", "log.warn", "log.error". Each message published by the producer will have the relevant routing key attached. You will then have a consumer that sends and email for all the error messages and another one that writes all the error messages to a file. So the emailer will define the binding from its queue to the exchange with the routing key "log.error". This way though the exchange receives all messages, the queue defined for the emailer will only contain the error messages. The filelogger will define a new separate queue bound to the same exchange and set up a different routing key. You could do three separate bindings for the three different routing keys require or just use the wildcard "log.*" to request all messages from the exchange starting with log.
This is a simple example that shows how you can achieve what you want to do.
look here for code examples specifically number tutorial number 5.

Making the best of exchange/routing of rabbitmq is recommended. If you do want to check according to the message content, the following code is a viable solution.
Retrieve messages from a queue and check, selectively ack the messages in which you're interested.
pull one message
GetResponse resp = channel.basicGet(QUEUE_NAME, false);
ack one message
channel.basicAck(resp.getEnvelope().getDeliveryTag(), false);
Example
import com.rabbitmq.client.*;
public class ReceiveLogs {
private final static String QUEUE_NAME = "hello";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
try(Connection connection = factory.newConnection();
Channel channel = connection.createChannel();){
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
// pull one message and ack manually and exit
GetResponse resp = channel.basicGet(QUEUE_NAME, false);
if( resp != null ){
String message = new String(resp.getBody(), "UTF-8");
System.out.println(" [x] Received '" + message + "'");
channel.basicAck(resp.getEnvelope().getDeliveryTag(), false);
}
System.out.println();
}
}
}
dependency
compile group: 'com.rabbitmq', name: 'amqp-client', version: '5.8.0'

To Retrieve Message from RabbitMQ we need to first connect with RabbitMQ server
public WebClient GetRabbitMqConnection(string userName, string password)
{
var client = new WebClient();
client.Credentials = new NetworkCredential(userName, password);
return client;
}
Now retrieve message from RabbitMQ using below code.
public string GetRabbitMQMessages(string domainName, string port,
string queueName, string virtualHost, WebClient client, string methodType)
{
string messageResult = string.Empty;
string strUri = "http://" + domainName + ":" + port +
"/api/queues/" + virtualHost + "/";
var data = client.DownloadString(strUri + queueName + "/");
var queueInfo = JsonConvert.DeserializeObject<QueueInfo>(data);
if (queueInfo == null || queueInfo.messages == 0)
return string.Empty;
if (methodType == "POST")
{
string postbody = "
{\"ackmode\":\"ack_requeue_true\",\"count\":
\"$totalMessageCount\",\"name\":\"${DomainName}\",
\"requeue\":\"false\",\"encoding\":\"auto\",\"vhost\" :
\"${QueueName}\"}";
postbody = postbody
.Replace("$totalMessageCount", queueInfo.messages.ToString())
.Replace("${DomainName}", domainName)
.Replace("${QueueName}", queueName);
messageResult = client.UploadString(strUri + queueName +
"/get", "POST", postbody);
}
return messageResult;
}
I think this will help you to implement RabbitMQ.

If you want to retrieve single message at a time please add the following properties with your Retrieving code .
Boolean autoAck = false;
model.BasicConsume(Queuename, autoAck);
model.BasicGet("Queuename", false);
model.BasicGet("Queuename", false);
By adding this properties of RabbitMQ you can retrieve the message one by one from the queue .Same like FIFO criteria

Related

Is there a way to programmatically delete an ActiveMQ Job Schedule?

I'm trying to delete a scheduled job in ActiveMQ, and so far no luck.
Schedule is created among with the message using either NMS API or Amqpnetlite (except openwire lib as that one is not updated and can't be used on netstandard/netcore)
Sample code used to create the schedule with NMS, same thing done with AMQP lib:
var factory = new Apache.NMS.ActiveMQ.ConnectionFactory(brokerUri);
IConnection connection = factory.CreateConnection(user, password);
connection.Start();
ISession session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
IDestination dest = session.GetQueue(destination);
IMessageProducer producer = session.CreateProducer(dest);
producer.DeliveryMode = MsgDeliveryMode.NonPersistent;
var msg = session.CreateTextMessage("Sample text message");
msg.Properties.SetString("AMQ_SCHEDULED_CRON", "* * * * *");
producer.Send(msg);
connection.Close();
This part produces the following result in browser console, and that's what I am willing to delete:
I've read this other question and answer, also active mq system constants but no way the schedule gets deleted. Also tried to browse over the documentation but can't find anything useful so far
Does ActiveMQ even supports a programmative way of managing schedules? An AMQP solution would be great, but NMS is also appreciated.
You can manage scheduled jobs in ActiveMQ over STOMP, AMQP or simply from a JMS client. I've written about this before showing how to do it using the ActiveMQ Java client but the principle is the same. You can send messages with specific headers set that will operate on the scheduled messages.
To browse the collection of scheduled message you'd do something like the following:
Connection connection = createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create the Browse Destination and the Reply To location
Destination requestBrowse = session.createTopic(ScheduledMessage.AMQ_SCHEDULER_MANAGEMENT_DESTINATION);
Destination browseDest = session.createTemporaryQueue();
// Create the "Browser"
MessageConsumer browser = session.createConsumer(browseDest);
connection.start();
// Send the browse request
MessageProducer producer = session.createProducer(requestBrowse);
Message request = session.createMessage();
request.setStringProperty(ScheduledMessage.AMQ_SCHEDULER_ACTION,
ScheduledMessage.AMQ_SCHEDULER_ACTION_BROWSE);
request.setJMSReplyTo(browseDest);
producer.send(request);
Message scheduled = browser.receive(5000);
while (scheduled != null) {
// Do something clever...
}
The returned messages contain the information on the actual scheduled message jobs that have been previously added. Obtaining the job Id allows you to completely cancel the delivery of said message.
To remove a scheduled message send that was scheduled using the Java client, AMQP client or other procotol client you'd do the following:
Message remove = session.createMessage();
remove.setStringProperty(ScheduledMessage.AMQ_SCHEDULER_ACTION,
ScheduledMessage.AMQ_SCHEDULER_ACTION_REMOVE);
remove.setStringProperty(ScheduledMessage.AMQ_SCHEDULED_ID,
scheduled.getStringProperty(ScheduledMessage.AMQ_SCHEDULED_ID));
producer.send(remove);
The full set of message property values that can be used when working with the scheduler are documented here, in AMQP just use the string literal of each as the Application Property value you set with the job Id to remote, or in the NMS client it'd just be a string key message property with the job Id that you want to remove.
There is one caveat though when doing this over AMQP and that would be that you'd need to ensure that the broker is using the JMS transformer ?transport.transformer=jms"see the AMQP documentation for ActiveMQ 5.
Accepted answer is working and correct. Turns out scheduledMessage.NMSMessageId was not holding the scheduler id.
Here's the clean C# source in case one is interested:
var factory = new Apache.NMS.ActiveMQ.ConnectionFactory(brokerUri);
IConnection connection = factory.CreateConnection(userName, password);
var session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
var requestBrowse = session.GetTopic("ActiveMQ.Scheduler.Management");
var queue = session.GetQueue(queueName);
var consumer = session.CreateConsumer(queue);
connection.Start();
var producer = session.CreateProducer(requestBrowse);
var scheduledMessage = consumer.Receive(TimeSpan.FromSeconds(10));
if (scheduledMessage != null)
{
// do check with persistent storage, if schedule is canceled, remove it:
var remove = session.CreateMessage();
// get prop names from : http://activemq.apache.org/maven/apidocs/constant-values.htm
remove.Properties["AMQ_SCHEDULER_ACTION"] = "REMOVE";
remove.Properties["scheduledJobId"] = scheduledMessage.Properties.GetString("scheduledJobId");
producer.Send(remove);
}
producer.Close();
session.Close();
connection.Close();
Flow is like this: Get messages from some queue, in case some condition is met, drop the schedule entirely.

Why does WCF ignore my TokenProvider?

I have a BizTalk WCF-Custom receive location to which I have added a custom behavior:
public class SasTokenProviderEndpointBehavior : BehaviorExtensionElement, IEndpointBehavior
{
public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
{
var tokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider(sharedAccessSecretName, sharedAccessKey);
bindingParameters.Add(new TransportClientEndpointBehavior { TokenProvider = tokenProvider });
}
}
}
parameter setup code omitted for brevity
This is adapted from a sample found at https://code.msdn.microsoft.com/How-to-integrate-BizTalk-07fada58#content - this author is widely respected in the BizTalk community and code of this kind has been in use for some years. All I am doing is adapting the method he uses, that is proven to work, to substitute a different TokenProvider.
I can see through debugging that this code runs and the TransportClientEndpointBehavior with correct parameters is added to the channel. However when the BizTalk receive location polls Service Bus, I see the following in the event log:
The adapter "WCF-Custom" raised an error message. Details "System.UnauthorizedAccessException: 40102: Missing authorization token, Resource:sb://[namespace].servicebus.windows.net/[queue]. TrackingId:452c2534-d3e6-400f-874f-09be324e9e11_G27, SystemTracker:[namespace].servicebus.windows.net:[queue], Timestamp:12/1/2016 11:38:56 AM ---> System.ServiceModel.FaultException: 40102: Missing authorization token, Resource:sb://[namespace].servicebus.windows.net/[queue]. TrackingId:452c2534-d3e6-400f-874f-09be324e9e11_G27, SystemTracker:[namespace].servicebus.windows.net:[queue], Timestamp:12/1/2016 11:38:56 AM
I cannot see any reason that the Azure Service Bus endpoint would return this error message except that because the token provider is not being used. Why would the channel ignore the TokenProvider and what do I have to do to pass the token correctly?
edit:
I have inspected the raw WCF message traffic for the port in question as well as one using the SB-Messaging adapter, which works as expected. The difference is that the SB-Messaging adapter's messages contain a SOAP header like:
<Authorization xmlns="http://schemas.microsoft.com/servicebus/2010/08/protocol/">SharedAccessSignature sr=[really long encoded string]</Authorization> and my custom binding port's messages do not. So it is true that the problem is a missing Authorization SOAP header; but the question persists - why isn't the channel adding this header?
edit #2:
I have decompiled Microsoft.ServiceBus.dll and I believe I've found the class that actually creates the WCF messsage, Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageCreator. It has this method:
private Message CreateWcfMessageInternal(string action, object body, bool includeToken, string parentLinkId, RetryPolicy policy, TrackingContext trackingContext, RequestInfo requestInfo)
{
Message message = Message.CreateMessage(this.messageVersion, action, body);
MessageHeaders headers = message.Headers;
headers.To = this.logicalAddress;
string sufficientClaims = this.GetSufficientClaims();
if (this.linkInfo != null)
{
if (!string.IsNullOrEmpty(this.linkInfo.TransferDestinationEntityAddress))
{
SecurityToken authorizationToken = this.GetAuthorizationToken(this.linkInfo.TransferDestinationEntityAddress, sufficientClaims);
if (authorizationToken != null)
{
SimpleWebSecurityToken webSecurityToken = (SimpleWebSecurityToken) authorizationToken;
if (webSecurityToken != null)
this.linkInfo.TransferDestinationAuthorizationToken = webSecurityToken.Token;
}
}
this.linkInfo.AddTo(headers);
}
if (includeToken)
{
ServiceBusAuthorizationHeader authorizationHeader = this.GetAuthorizationHeader(sufficientClaims);
if (authorizationHeader != null)
headers.Add((MessageHeader) authorizationHeader);
}
if (this.messagingFactory.FaultInjectionInfo != null)
this.messagingFactory.FaultInjectionInfo.AddToHeader(message);
if (!string.IsNullOrWhiteSpace(parentLinkId))
message.Properties["ParentLinkId"] = (object) parentLinkId;
if (trackingContext != null)
TrackingIdHeader.TryAddOrUpdate(headers, trackingContext.TrackingId);
MessageExtensionMethods.AddHeaderIfNotNull<RequestInfo>(message, "RequestInfo", "http://schemas.microsoft.com/netservices/2011/06/servicebus", requestInfo);
return message;
}
So thinking about it logically, there are two reasons the Authorization header would be missing:
includeToken is false (Why would this be so?)
GetAuthorizationHeader() returns null (Why?)
edit #3:
I have compiled and run the example code and this works. The only significant difference between my code and his is that mine includes a line which calls out to Azure Key Vault:
var kv = new KeyVaultClient(this.GetAccessToken);
var key = kv.GetSecretAsync(this.KeyVaultUri.AbsoluteUri, this.SharedAccessSecretName).Result;
var sharedAccessKey = key.Value;
var tokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider(
this.SharedAccessSecretName,
sharedAccessKey);
bindingParameters.Add(new TransportClientEndpointBehavior { TokenProvider = tokenProvider });
This is an asynchronous method that returns a Task. Can it be that blocking on the result of this Task somehow doesn't do what would be expected in certain situations, and this is messing up the configuration of the WCF channel somehow? As I said, I am certain this code runs and assigns the TokenProvider. I am now merely not certain when it runs.
D'OH!
I had neglected to realise that the very old version of Microsoft.ServiceBus.dll we still have in the solution for interop with the (equally old) on premises version of Service Bus (Service Bus for Windows Server) was the one referenced by my project. For whatever reason this version just doesn't do what it's supposed to, and doesn't give any indication that it's bypassing the intended behaviour. Updating to have the current NuGet package for Service Bus fixes the problem.

Is there support for the RabbitMQ mandatory flag in MassTransit

I would like to get an exception when sending to a RabbitMQ exchange using MassTransit if the message is not delivered to any queues. It looks like the RabbitMQ mandatory flag could be used for something like this, but I have not been able to find any way to set this using MassTransit. Is this supported?
My code:
var personSendEndpoint = busControl.GetSendEndpoint(new Uri($"rabbitmq://localhost/{personQueueName}")).Result;
// An exception should occur on the following call if the message is not delivered to any queue
personSendEndpoint.Send(new PersonUpdated() { Name = "Dorian Gray", Key = Guid.NewGuid() });
You can set the Mandatory flag, but there isn't a cute extension methods to do it for you. Basically, you do as shown below:
await endpoint.Send(new PersonUpdate(...), context =>
{
(context as RabbitMqSendContext).Mandatory = true;
});
And that will set the flag telling RabbitMQ to throw an exception if no bindings for the exchange exist.

RabbitMQ: how to delete exchanges in a batch?

I have created a lot of error exchanges (using EasyNetQ), about 1400 items. The name of each exchange looks like ErrorExchange_xxxxx.
I want to delete them all.
Is there any script to delete the exchanges in a batch?
There is a method exchangeDelete available with Channel API
You can write a method such as below which takes list of exchangeNames to be deleted :
DELETE EXCHANGE
public void deleteExcahnges(List<String> exchangeNames){}
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(localhost);
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
for(String exchangeName : exchangeNames){
channel.exchangeDelete(exchangeName);
}
}
It would delete all the exchanges.
Note:-> I am using amqp-client.jar for connecting to RabbitMQ server

Redis publish to one client

Is it possible to publish a message to only one client subscribed to channel? I know how to get the address (client list). Is it possible to get the address and publish to only that client?
In Redis you publish to a channel, not to a subscriber.
So the only way should be having one channel per client.
You can probably have one common channel to share the address, and then use the addresses as the channels name.
http://redis.io/topics/pubsub
As an example, using C# and StackExchange.Redis library:
//SERVER
var cnn = ConnectionMultiplexer.Connect("localhost");
var sub = cnn.GetSubscriber();
sub.Subscribe("common", (channel, value) =>
{
Console.WriteLine("SERVER Received in common: {0}", value);
sub.Publish("channel:" + value, "private from server to " + value);
});
//CLIENT
var cnn = ConnectionMultiplexer.Connect("localhost");
var address = "client1";
var sub = cnn.GetSubscriber();
sub.Subscribe("channel:" + address, (channel, value) =>
{
Console.WriteLine("Private: {0}", value);
});
sub.Publish("common", address);
Another solution: what I was trying to do was to have all subscribers listen but only one handle the message. The way I did this was create a stack in redis and have all subscribe to a key's list command notifications. So I would RPUSH a new value onto the key, the subscribers would get the keyspace notification and the fist to LPOP the value would handle it.